Subject: v06i053: Usenet news batcher control program (newbatchB) Newsgroups: mod.sources Approved: rs@mirror.UUCP Submitted by: Rich $alz for Chris Lewis Mod.sources: Volume 6, Issue 53 Archive-name: newbatchB [ This is the second of two batched news utility programs (BNUP, to use an acronym). It was written on a BSD4.3 machine, but it looks like it would be fairly easy to convert to USG. Chris Lewis had posted this to the net shortly before he left it, and I cleaned up the documentation and am publishing it here. Both this program and the previous one have some nice features; I look forward to the next one that combines the best of both methods, portably. --r$ ] #!/bin/sh # This is a shell archive. Remove anything before this line, # then unpack it by saving it in a file and typing "sh file". # Contents: README batcher.1 batcher.ctrl batcher.c Makefile echo x - README sed 's/^XX//' > "README" <<'@//E*O*F README//' XXThis program is a ``posthumous'' publication of softare Chris Lewis XXposted to the net. I felt it was worthwhile, and decided to clean up XXthe documentation a bit and publish it in mod.sources. This README, XXthe manpage, and the sample control file come from Chris's original XXposting, combined with additional information I gleaned from the source. XXRead carefully -- there might be errors in the documentation. XX /Rich $alz, XX Moderator of mod.sources XXNews admins! Sick of having millions of cron entries, differing XX[cs7]sendbatch scripts, and blown out spool areas with your downstream XXfeeds? Well, have I got something for you! XXThe "batcher" program which works with 2.10.2 and 2.10.3 news, solves XXall of these problems (if your're lucky!). You install it in XX/usr/lib/news along with a batch control file, "batcher.ctrl". The XXcontrol file contains one line per down-stream site which describes how XXto send news to that site. XXTo install, read these next few paragraphs and edit the source as XXappropriate. XX +Batching instructions for all downstream sites is contained in one XX file, instead of having to make customized *sendbatches (e.g., XX differing bits in compress, different grades, different batch XX sizes). XX +You can speicfy the batch size for each site, tuning for various XX link qualities. XX +If you have 4.3BSD UUCP, you can completely eliminate the chance XX of filling up your spool area. The control file lets you specify XX the number of bytes allowable in the UUCP queue for each system; XX the real batching is not invoked for a site until they make contact XX and bring the queue size back down below the limit. XX This is done by 'popen("uuq -l -s", "r")' to find out the XX size of the queue for the site. Non-4.3 sites can probably write XX their own routine to scan UUCP "C." files to figure out how big the XX queue is for a specific system. If you don't have this uuq command, XX #undef UUQ in the source. XX +If you are running on a 4.xBSD system, batcher will also not invoke XX the real batching software if there is less than 5Mb free on the XX UUCP spool file system. This is done by doing "df /usr/spool/uucp" XX which I believe works only on 4.[23]BSD systems. If you can't do XX this, #undef DFABLE in the source. It should be fairly easy to XX kludge something up for other sites, however. XXNext, modify your news sys file so that each system you want to control XXthis way has batching turned on: XX mnetor:....:F:/usr/spool/batch/mnetor XXAdd entries for those sites in /usr/lib/news/batcher.ctrl. Information XXon the control file is given in the manpage, and in the sample file XXdistributed. XXFinally, via cron, run batcher setuid news with either the name of a XXsite to feed, or a "class" (which'll run all sites with that class). XXThis software has made an incredible difference on this site. We give XXfull feeds to 7 sites, plus smaller ones to about another ten. Until XXthis was installed, we had to disable batching to one site or another XXevery second day. And we blew spool areas on weekends. We haven't had XXto touch *anything* since this was installed. Even when sites have XXbeen unreachable for weeks on end. XXChris Lewis XX formerly at mnetor!clewis, XX might now be reachable at utzoo!spectrix!clewis @//E*O*F README// chmod u=rw,g=rw,o=rw README echo x - batcher.1 sed 's/^XX//' > "batcher.1" <<'@//E*O*F batcher.1//' XX.TH BATCHER 1 LOCAL XX.SH NAME XXbatcher \- a new batching system for news XX.SH SYNOPSIS XX.B batcher XX[ XX.IB \-c X XX] XX.br XX.B batcher XX[ XX.I site XX] XX.IR "(typically run out of " cron ".)" XX.SH DESCRIPTION XX.I Batcher XXis designed to feed batched USENET neighbors in a controlled, orderly fashion. XXIn particular, it is designed to replace multiple [cs7]sendbatch scripts, XXand put an upper limit on the amount of data that may be queued for a site. XXAt some sites, it is also possible to perform checks that will prevent XXoverflowing the filesystem that contains the UUCP spool directory. XX.PP XX.I Batcher XXreads the file XX.RI NEWSLIB/batch.ctrl XXto determine how to send data to each site. XXThis control file contains one line per down\-stream site that describes how XXto send news to that site. XX.I Batcher XXitself is run out of XX.I cron XXas necessary, setuid news. XXIt should be given a single argument, either the name of a site to batch the XXnews for, or a class XX.RI ( batcher XXwill then do work on all sites in that class). XX.PP XXIn the control file, lines beginning with a pound sign (``#'') are ignored. XXData lines, one per site being fed, are comprised of seven colon\-separated XXfields: XX.RS XXsite:class:compress:uux_flags:b_size:q_size:uux_command XX.RE XX.TP site XXThe name of the site being fed. This is the same as the name in the news XX.I sys XXfile. XXBy using the ``F'' protocol in news, the file XX.IR /usr/spool/batch/ site XXshould contain the full\-pathname list of articles intended for that site. XX.TP class XXA single letter; the default is ``z.'' Executing ``batcher \-cX'' works on all XXsites in class ``X.'' XX.TP compress XXThis is the command line to use to call XX.IR compress ; XXthe default is no compression. XXIf something is specified here, it is used as an intermediate pipe, and the XX``uux'' field defaults to ``uux\ ...\ !cunbatch.'' XX.TP uux_flags XXThese are the flags to insert on the uux command line. As distributed, he XXdefault is ``\-r\ \-z\ \gd'' (just queue work, only acknowledge failures, XXgrade d transmission). XX.TP b_size XXThe size of batches, default is 100K. XXThis is passed on to the standard XX.I batcher XXprogram. XX.TP q_size XXThe maximum UUCP queue size, default is 500K. As distributed, the program XXparses the output of the 4.3BSD command ``uuq\ \-l\ \-sSITE.'' XXIf the queue is currently greater than ``q_size,'' no work is done for XXthe site until it drops below that number. XX.TP uux_command XXThis can be a complete replacement for the uux command. The default is XX``uux\ \-\ \ site!.'' XX.PP XXTo use XX.IR batcher , XXmodify your news XX.I sys XXfile so that all (most?) (some??) sites use the batching protocol, e.g.: XX.RS XXmnetor:net.flame,net.rumor,net.bizarre:F:/usr/spool/batch/mnetor XX.RE XXNote that the batch filename must be the same name as the site being fed. XXNext edit the control file, placing sites in classes and with queue sizes XXthat seem appropriate. XXThen, edit XX.I /usr/lib/crontab XXto replace calls to the old batching programs with calls to XX.IR nbatcher . XX(It should be running setuid news.) XXYou will probably want to use both ``job class'' calls and explicitly\-named XXsites. XX.SH BUGS XXQueue and spool\-fillage checking is done via XX.I uuq XXand XX.IR df , XXrespectively. This is somewhat slow, but it is swamped by the execution XXof the batching itself. @//E*O*F batcher.1// chmod u=rw,g=rw,o=rw batcher.1 echo x - batcher.ctrl sed 's/^XX//' > "batcher.ctrl" <<'@//E*O*F batcher.ctrl//' XX# SAMPLE BATCHER.CTRL FILE XX# XX# Lines beginning with pound signs are ignored. Data lines look like; XX# site:class:compress:uux_flags:b_size:q_size:uux_command XX# Where XX# Site = Name of downstream neighbor XX# Class = System class; "batcher -cX" does all sites in class X. XX# Default class is z. XX# Compress = Anything here is intermediate pipe to the uux, also XX# changes default uux_command (see below). XX# Uux_flags = Put in uux command if not given in field seven. XX# Default is "-r -z -gd" (see below). XX# B_size = Size of batches; default is 100K. XX# Q_size = Size of UUCP queue; no work is done while # bytes XX# in queue is greater than this. XX# Uux_command = Complete replacement for uux command. Default is XX# "uux - site!" (see XX# above). XX# If a name does not exist, batcher won't send a batch. XX# XX# SAMPLE ENTRIES XX# XX# NORMAL NEWS FEEDS XXgenat::compress -C:::: XXutcs::compress -C:::: XXradha::compress:::: XXtoram::compress:-n -r -gd::: XXyetti:z:compress -C:::: XXlsuc::compress -C -b 13:::: XXmicomvax:::::: XX# XX# NEW NEWS FEED COMING UP XXsyntron::compress -C:::: XX# XX# MOT NEWS FEEDS XXmotsj1:A::-n -r -gd::: XXmot:A::-n -r -gd::: XX# XX# "-cX" NEWS FEEDS XXcxhq:A::-n -gB::: XXcxsea:A::-n -gB::: XXcxmso:A:compress -C:-n -gB::: XXcxsch:A:::20000::mail cxsch!ewa XXcxphx:A:::20000::mail cxphx!nms XXcxmd:A:::20000::mail cxmd!joe XXtest:A:::::mail clewis @//E*O*F batcher.ctrl// chmod u=rw,g=rw,o=rw batcher.ctrl echo x - batcher.c sed 's/^XX//' > "batcher.c" <<'@//E*O*F batcher.c//' XX/* Chris Lewis, June 1986 */ XX#include XX#if defined(BSD4_2) || defined(BSD4_1C) XX#include XX#else XX#include XX#endif XXextern char *malloc(); XXextern void free(); XX#define NUMFLDS 7 XX#define NAME 0 XX#define CLASS 1 XX#define COMPRESS 2 XX#define UUXFLAGS 3 XX#define SIZE 4 XX#define QSIZE 5 XX#define UUX 6 XX#define UUQ /* define this if you have BSD 4.3 uuq utility */ XX#define DFABLE /* define this if "df /usr/spool/uucp" works on your system */ XX/* Take a look at these, too. */ XX#ifdef DEBUG XX#define BATCHCTRL "batcher.ctrl" XX#else XX#define BATCHCTRL "/usr/lib/news/batcher.ctrl" XX#endif XX#define BATCH "/usr/spool/batch" XX#define SBATCH "/usr/lib/news/batch" XXint fldlen[NUMFLDS] = {10, 2, 20, 10, 7, 10, 30}; XXstruct desc { XX char *flds[NUMFLDS]; XX struct desc *next; XX} *head = (struct desc *) NULL, XX *dptr = (struct desc *) NULL; XX#if defined(BSD4_2) || defined(BSD4_1C) XX#define strchr index XX#endif XXstruct desc *getflds(); XXint verbose = 0; XXint class; XXint spoolok = 1; XXlong spoollim = 5000000; XX/* XX * main: XX * - process arguments XX * - read control file XX * - for each system selected, see if there is any work, XX * if so, go try to do it. XX */ XXmain(argc, argv) XXint argc; XXchar **argv; { XX register char *p; XX register struct desc *curptr; XX argc--; argv++; XX for (;argc > 0 && **argv == '-'; argv++) { XX for (p = (*argv)+1; *p; p++) XX switch(*p) { XX case 'v': XX verbose = 1; XX break; XX case 'c': XX class = *(p+1); XX if (class == 0) XX class = 'z'; XX else XX p++; XX break; XX default: XX fprintf(stderr, "batcher: Bad arg %s\n", *argv); XX exit(1); XX } XX } XX readctrl(); XX if (!checkspool()) { XX exit(0); XX } XX if (verbose) XX dumpctrl(); XX if (class) { XX for (curptr = head; curptr && spoolok; curptr = curptr->next) { XX if (*(curptr->flds[CLASS]) == class && XX (chkbatch(curptr->flds[NAME], "") || XX chkbatch(curptr->flds[NAME], ".work")) && XX (spoolok = checkspool())) XX doit(curptr->flds[NAME]); XX } XX } XX while(*argv && spoolok) { XX if (chkbatch(*argv, "") || chkbatch(*argv, ".work")) { XX doit(*argv); XX } XX argv++; XX } XX exit(0); XX} XX/* XX * readctrl: XX * XX * Each line in the batch.ctrl file contains NUMFLDS colon-separated XX * parameters. This function reads each line, and calls getflds XX * to separate out the parameters. If getflds returns a system descriptor XX * it is linked into the list of system descriptors. XX */ XXreadctrl() { XX char buf[BUFSIZ]; XX register char *p; XX register struct desc *ptr; XX struct desc *getflds(); XX register FILE *ctrl = fopen(BATCHCTRL, "r"); XX if (!ctrl) { XX fprintf(stderr, "batcher: could not open %s file\n", XX BATCHCTRL); XX exit(1); XX } XX while (fgets(buf, sizeof(buf), ctrl)) { XX if (buf[0] == '#') XX continue; XX p = buf + strlen(buf) - 1; XX if (*p == '\n') XX *p = '\0'; XX if ((ptr = getflds(buf)) && processctrl(ptr)) { XX if (!head) XX head = dptr = ptr; XX else { XX dptr->next = ptr; XX dptr = ptr; XX } XX ptr->next = (struct desc *) NULL; XX } XX } XX fclose(ctrl); XX} XX/* XX * dumpctrl: XX * XX * If verbose is on, dump the tables XX */ XXdumpctrl() { XX register struct desc *p; XX register int i; XX for (p = head; p; p = p->next) { XX for (i = 0; i < NUMFLDS; i++) XX printf("%-*s", fldlen[i], p->flds[i]); XX printf("\n"); XX } XX} XXchar *strsave(); XX/* XX * getflds: XX * XX * This routine parses a single line from the batch.ctrl file, XX * and if successfully parsed and checked out, returns a system XX * descriptor pointer XX */ XXstruct desc * XXgetflds(buf) XXchar *buf; { XX register int cnt; XX char b2[100]; XX char *curp, *p; XX int len; XX struct desc *dptr; XX dptr = (struct desc *) malloc(sizeof (struct desc)); XX if (!dptr) { XX fprintf(stderr, "batcher: Cannot malloc\n"); XX exit(1); XX } XX curp = buf; XX for (cnt = 0; cnt < NUMFLDS; cnt++) { XX if (cnt == (NUMFLDS - 1)) { XX if (strchr(curp, ':')) { XX fprintf(stderr, "batcher: too many colons: %s\n", XX buf); XX free(dptr); XX return(NULL); XX } XX p = curp + strlen(curp); XX } else XX p = strchr(curp, ':'); XX if (p == NULL) { XX fprintf(stderr, "batcher: invalid format (%d): %s\n", XX cnt, buf); XX free(dptr); XX return(NULL); XX } XX len = p - curp; XX if (len >= fldlen[cnt]) { XX fprintf(stderr, "batcher: field %d too long: %s\n", XX cnt+1, buf); XX free(dptr); XX return(NULL); XX } XX if (!(dptr->flds[cnt] = malloc(len + 1))) { XX fprintf(stderr, "batcher: cannot malloc\n"); XX exit(1); XX } XX strncpy(dptr->flds[cnt], curp, len); XX dptr->flds[cnt][len] = '\0'; XX curp = p + 1; XX } XX return(dptr); XX} XX/* XX * strsave: XX * returns pointer to malloc'd copy of argument XX */ XXchar * XXstrsave(s) XXchar *s; { XX register char *p = malloc(strlen(s) + 1); XX if (!p) { XX fprintf(stderr, "batcher: cannot malloc\n"); XX exit(1); XX } XX strcpy(p, s); XX return(p); XX} XX/* XX * chkbatch: XX * XX * return 1 if a batcher work file / exists. XX */ XXchkbatch(name, type) XXchar *name; XXchar *type; { XX char batch[BUFSIZ]; XX sprintf(batch, "%s/%s%s", BATCH, name, type); XX if (access(batch, 04) == 0) XX return(1); XX else XX return(0); XX} XX/* XX * doit: XX * XX * This routine is called with the name of the system that has XX * been determined to have work for it. The system is searched XX * for in the system descriptors. If found, a "system" line XX * is contructed from the tables, and executed if system has not XX * exceeded it's UUCP queue limit. XX */ XXdoit(name) XXchar *name; { XX char cmdbuf[BUFSIZ]; XX int rc; XX long queuesize; XX long checkqueue(), checkspool(); XX long queuemax; XX register struct desc *ptr; XX if (verbose) XX printf("batcher: doing %s\n", name); XX for (ptr = head; ptr; ptr = ptr->next) XX if (!strcmp(ptr->flds[NAME], name)) { XX /* form the command line for batching */ XX sprintf(cmdbuf, "%s %s/%s %s", XX SBATCH, BATCH, name, ptr->flds[SIZE]); XX if (*(ptr->flds[COMPRESS])) XX sprintf(cmdbuf + strlen(cmdbuf), "|%s", XX ptr->flds[COMPRESS]); XX /* Find the queue limit */ XX sprintf(cmdbuf + strlen(cmdbuf), "|%s", ptr->flds[UUX]); XX if (1 != sscanf(ptr->flds[QSIZE], "%ld", &queuemax)) { XX fprintf(stderr, "batcher: bad qmax field: %s\n", XX ptr->flds[QSIZE]); XX return; XX } XX queuesize = checkqueue(ptr->flds[NAME]); XX rc = 0; XX /* While we haven't exceeded the queue limit & XX there's work to do, issue the command */ XX while (queuesize < queuemax && !rc && XX (chkbatch(ptr->flds[NAME], "") || XX chkbatch(ptr->flds[NAME], ".work")) && XX (spoolok = checkspool())) { XX#ifdef DEBUG XX printf("batcher: cmd: %s\n", cmdbuf); XX rc = 1; XX#else XX rc = system(cmdbuf); XX queuesize = checkqueue(ptr->flds[NAME]); XX#endif XX } XX return; XX } XX fprintf(stderr, "batcher: no control line for %s\n", name); XX} XX/* XX * processctrl: XX * XX * Check validity of batch.ctrl entries and supply defaults. XX */ XXprocessctrl(ptr) XXstruct desc *ptr; { XX char buf[100]; XX register char *p, *uuxflags; XX if (!ptr) return; XX if (strlen(ptr->flds[NAME]) == 0) { XX fprintf(stderr, "batcher: null system name\n"); XX free(ptr); XX return(0); XX } XX if (strlen(ptr->flds[QSIZE]) == 0) { XX free(ptr->flds[QSIZE]); XX ptr->flds[QSIZE] = strsave("500000"); XX } XX if (strlen(ptr->flds[CLASS]) == 0) { XX free(ptr->flds[CLASS]); XX ptr->flds[CLASS] = strsave("z"); XX } XX if (strlen(ptr->flds[SIZE]) == 0) { XX free(ptr->flds[SIZE]); XX ptr->flds[SIZE] = strsave("100000"); XX } XX if (strlen(ptr->flds[UUXFLAGS]) == 0) { XX free(ptr->flds[UUXFLAGS]); XX ptr->flds[UUXFLAGS] = strsave("-r -z -gd"); XX } XX if (strlen(ptr->flds[UUX]) == 0) { XX sprintf(buf, "uux - %s %s!%s", ptr->flds[UUXFLAGS], XX ptr->flds[NAME], XX *ptr->flds[COMPRESS] ? "cunbatch" : "rnews"); XX ptr->flds[UUX] = strsave(buf); XX } XX return(1); XX} XX/* XX * checkqueue: XX * XX * Logically, all this code does is return the size of the UUCP queue XX * for system "name". XX * I've taken the easy way out and popen'd "uuq" (4.3 BSD UUCP utility) XX * to parse the first line, which looks something like this: XX * XX * : jobs, bytes, .... XX * XX * I merely look for the first comma, and sscanf the number following. XX * A proper solution would be to dive in and parse the UUCP queues XX * themselves, but: it's moderately difficult, and it changes from XX * system to system. XX */ XXlong XXcheckqueue(name) XXchar *name; { XX#ifdef UUQ XX char buf[BUFSIZ]; XX long retval; XX register char *p2; XX FILE *p, *popen(); XX /* Gross, but the easiest way */ XX sprintf(buf, "uuq -l -s%s", name); XX p = popen(buf, "r"); XX if (!fgets(buf, sizeof(buf), p)) { XX return(0); XX } XX pclose(p); XX p2 = strchr(buf, ','); XX if (p2 && 1 == sscanf(p2+1, "%ld", &retval)) { XX return(retval); XX } XX fprintf(stderr, "batcher: could not interpret %s\n", buf); XX return(10000000); XX#else XX return(10000000); XX#endif XX} XX/* This function returns the amount of free space on the spool XX device, this may not work on your system - it reads the XX second line from a "df /usr/spool/uucp" */ XX#define SPOOL "/usr/spool/uucp" XXcheckspool() { XX#ifdef DFABLE XX char buf[100]; XX FILE *p, *popen(); XX long val; XX sprintf(buf, "df %s", SPOOL); XX if (!(p = popen(buf, "r"))) { XX fprintf(stderr, "batcher: couldn't popen %s\n", buf); XX return(0); XX } XX if (!fgets(buf, sizeof(buf), p)) { XX fprintf(stderr, "batcher: no first line in df\n"); XX return(0); XX } XX if (!fgets(buf, sizeof(buf), p)) { XX fprintf(stderr, "batcher: no second line in df\n"); XX return(0); XX } XX if (1 != sscanf(buf, "%*s %*ld %*ld %ld", &val)) { XX fprintf(stderr, "batcher: couldn't get size from %s\n", buf); XX return(0); XX } XX if (pclose(p)) { XX fprintf(stderr, "batcher: DF failed\n"); XX return(0); XX } XX val *= 1024; XX if (val < spoollim) { XX printf("batcher: spool limit exceeded, %ld bytes left\n", val); XX return(0); XX } else XX return(1); XX#else XX return(1); XX#endif XX} @//E*O*F batcher.c// chmod u=rw,g=rw,o=rw batcher.c echo x - Makefile sed 's/^XX//' > "Makefile" <<'@//E*O*F Makefile//' XXDEFS=-DBSD4_2 XXCFLAGS = -O $(DEFS) XXbatcher: batcher.c XX cc -o batcher $(CFLAGS) batcher.c @//E*O*F Makefile// chmod u=rw,g=rw,o=rw Makefile echo Inspecting for damage in transit... temp=/tmp/sharin$$; dtemp=/tmp/sharout$$ trap "rm -f $temp $dtemp; exit" 0 1 2 3 15 cat > $temp <<\!!! 69 533 3297 README 102 557 3260 batcher.1 46 209 1374 batcher.ctrl 441 1376 9503 batcher.c 4 12 88 Makefile 662 2687 17522 total !!! wc README batcher.1 batcher.ctrl batcher.c Makefile | sed 's=[^ ]*/==' | diff -b $temp - >$dtemp if test -s $dtemp then echo "Ouch [diff of wc output]:" ; cat $dtemp else echo "No problems found." fi exit 0