Subject: v19i098: Usenet sources archiver, Part01/04 Newsgroups: comp.sources.unix Sender: sources Approved: rsalz@uunet.UU.NET Submitted-by: Kent Landfield Posting-number: Volume 19, Issue 98 Archive-name: rkive/part01 rkive reads a configuration file to determine such things as: o where the news directory resides, o where each newsgroup is to be archived, o the type of archiving to be done for each newsgroup, o the ownership and modes of the archived members, as well as additional optional features such as: o which users/accounts to mail the archived member information to, o the location and format of log files, o the location and format of index files, o the compression program to use (if desired). It is intended that rkive be run by cron on a daily basis. In this manner, software is archived and available for retrieval from the archives on the day it reaches the machine. It allows for the archives to be managed by the same or different people (or accounts). It supports the building of indexes for later review or to interface to the netlib type of mail retrieval software. It also supports mailing notifications of the archiving to a specified list of users or aliases. The indexes and log file formats are specifiable by the person configuring the rkive configuration file. This package was initially designed for archiving comp.sources.all newsgroups. It does however, support archiving of non-moderated, non-sources newsgroups. #! /bin/sh # This is a shell archive. Remove anything before this line, then unpack # it by saving it into a file and typing "sh file". To overwrite existing # files, type "sh file -c". You can also feed this as standard input via # unshar, or by typing "sh 'IDEAS' <<'END_OF_FILE' X@(#)IDEAS 1.1 6/1/89 XWhat follows is my ideas file. It gives you a glimse into some enhancements, Xideas, problems and work in progress for rkive. This is a minimal list but Xit will grow after you send me your good ideas. :-) X X1. If a posting should replace the current entry it is the moderator's X responsibility to Supersedes: any invalid postings. rkive should X be able to handle Supersedes. Currently it does not. X X2. Add Support For Multiple Archive Formats. X Have rkive link the files into a different archiving format. X All articles should be storable in *as many* of the user-requested X formats as possible. X X3. Add internal method of numbering the articles. A *new* method of archiving. X If the archive is moved to a different machine running news or if the X news subsystem is restarted from scratch on the same machine, it could X conflict with previously archived articles in archives that use X Article-Number, thus producing massive amounts of duplicate archive X files in which the only problem is the file names. X X4. Add a Configure.sh to handle configuration of default parameters X and to setup the software. X X5. Better and more extensive documentation as well as documentation X that explains archiving in general. X X6. Remote archive BASEDIRs. This would allow the archive to be scattered X over multiple machines connected via a LAN. With the use of rsh, rcp, X etc., this should not be too hard. It would require the configuration X file usage to be expanded so that the admin could specify the method X for storing. (builtin versus rcp versus user supplied program) The X BASEDIR variable usage would also have to be expanded so that it X understood "machine:directory-path" formatting. BEWARE of security X concerns here.... :-( A generic hook to user-defined storage X programs/libraries. I am wide open to ideas here... :-) X X7. An application that would allow retrieval requests from the archives X by reading the rkive.cf file to determine the location of the X newsgroup's archive and to retrieve packages by any of the archive X methods. This may end up using software like narc to actually do X the unpacking for the requester. This too is just an idea now... X X8. An application that would allow retrieval requests from the archives X for patches or entire packages including all posted patches. This is X currently in progress... :-) The mythical "random downloader" :-) X XWell that's the direction I am heading... If you have *any* additional Xideas that are constructive, positive or negative (flames to /nev/dull) XI will be glad to hear from you. X X Good luck and Happy archiving...:-) X kent@ssbell X END_OF_FILE if test 2729 -ne `wc -c <'IDEAS'`; then echo shar: \"'IDEAS'\" unpacked with wrong size! fi # end of 'IDEAS' fi if test -f 'MANIFEST' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'MANIFEST'\" else echo shar: Extracting \"'MANIFEST'\" \(1985 characters\) sed "s/^X//" >'MANIFEST' <<'END_OF_FILE' X File Name Archive # Description X----------------------------------------------------------- X IDEAS 1 Contains future improvement ideas. X MANIFEST 1 This shipping list X Makefile 2 rkive make file for generating the software. X README 1 Information posting that should be read first. X article.1 2 Manual page for article command. X article.c 2 Source containing article main routines. X article.h 1 Include file for News article information. X cfg.h 1 Include file for variable declarations. X ckconfig.1 1 Manual page for ckconfig command. X ckconfig.c 2 Source containing ckconfig main routine. X disp_grp.c 1 Display newsrgoup configuration file info.. X efopen.c 1 Fopen a file with error checking. X format.c 2 Formatting output for indexes, logs, articles. X header.c 3 News article header reading routines. X makedir.c 1 Make directory routines. X news_arc.c 4 News archiving gut functions. X patchlevel.h 1 Include file containing current patchlevel. X record_arc.c 2 Routines to deal with .archived file. X rename.c 1 Routine to rename a file. X rkive.1 1 Manual page for the rkive command. X rkive.5 2 Manual page for the rkive configuration file. X rkive.c 4 Source containing rkive main routines. X rkive.cf 3 Template configuration file. X rkive.h 2 Include file for rkive software. X setup.c 3 Routines to read the rkive configuration file. X str.c 1 String manipulation routines. X t.cf 1 Test archive config file. Not the template. X version.c 1 Print the current version and patchlevel. END_OF_FILE if test 1985 -ne `wc -c <'MANIFEST'`; then echo shar: \"'MANIFEST'\" unpacked with wrong size! fi # end of 'MANIFEST' fi if test -f 'README' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'README'\" else echo shar: Extracting \"'README'\" \(14704 characters\) sed "s/^X//" >'README' <<'END_OF_FILE' X X USENET Sources Archiver X X @(#)README 1.1 6/1/89 X X Copyright (c) 1989, by Kent Landfield. X X Permission is hereby granted to copy, distribute or otherwise X use any part of this package as long as you do not try to make X money from it or pretend that you wrote it. The copyright X notice must be maintained in any copy made. X X If you make modifications to this software that you feel X increases it usefulness for the rest of the community, please X email the changes, enhancements, bug fixes as well as any and X all ideas to me. This software is going to be maintained and X enhanced as deemed necessary by the community. X X -Kent+ X uunet!ssbell!kent X X------------------------------------------------------------------ X DISCLAIMER X------------------------------------------------------------------ XUse of this software constitutes acceptance for use in an AS IS Xcondition. There are NO warranties with regard to this software. XIn no event shall the author be liable for any damages whatsoever Xarising out of or in connection with the use or performance of this Xsoftware. Any use of this software is at the user's own risk. X------------------------------------------------------------------- X X XWhen made, this package currently contains 3 executables: X X o rkive - a USENET newsgroup archiver, X o article - print formatted news article header information, and X o ckconfig - an rkive configuration file check program. X X XThis package was initially designed for archiving comp.sources.all newsgroups. XIt does however, support archiving of non-moderated, non-sources newsgroups. X X X ----- X rkive X ----- X Xrkive reads a configuration file to determine such things as: X X o where the news directory resides, X o where each newsgroup is to be archived, X o the type of archiving to be done for each newsgroup, X o the ownership and modes of the archived members, X Xas well as additional optional features such as: X X o which users/accounts to mail the archived member information to, X o the location and format of log files, X o the location and format of index files, X o the compression program to use (if desired). X XIt is intended that rkive be run by cron on a daily basis. In this manner, Xsoftware is archived and available for retrieval from the archives on the Xday it reaches the machine. It allows for the archives to be managed by Xthe same or different people (or accounts). It supports the building Xof indexes for later review or to interface to the netlib type of mail Xretrieval software. It also supports mailing notifications of the archiving Xto a specified list of users or aliases. The indexes and log file formats Xare specifiable by the person configuring the rkive configuration file. X X------------------------------------------------------------------- XThe following defines are possible. Please note that the Directory XCreation defines are specified in Makefile while the rest are specified Xin rkive.h. X X*************************** Xrkive.h - General Defines X*************************** X X-D REDUCE_HEADERS : Archived article header reduction code. X Disk space is saved by removing header lines that have no X further use after the article is stored in the archive. X As currently defined, all headers *except* for From:, Newsgroups:, X Subject:, Message-ID: and Date are removed if this is defined. X The list of headers to be saved can be added to or reduced by X modifying the table "hdrs" in news_arc.c. X X-D SUBJECT_LINE : Specify that the local mailer has -s option X such as /usr/bin/mailx or /usr/ucb/Mail. X X************************************* XMakefile - Directory Creation Defines X************************************* X X-D HAVE_MKDIR : use the mkdir() function in the system library. X (AT&T 5.2 or earlier systems are probably out of luck..) X X-D USE_SYSMKDIR : have rkive system off /bin/mkdir. X (not recommended for *real* use...) X XIf you do not define either, the function makedir() will create the Xdirectory itself. I suggest that if you do not have mkdir() in your Xsystem libraries, use the builtin if you can. *Please* verify you can Xuse it *first*. X X--------------------------- XArchive Member Compression: X--------------------------- XIf you wish to have your archived articles compressed you may do so by Xspecifying the disk path to the compression program as the value for XCOMPRESS in the rkive configuration file. It is important that *if* you Xuse a compression program other that "compress" or "pack" that you add Xa an entry to the compression routine table just above the function Xsuffix() in news_arc.c. Currently, this program recognizes just ".z" and X".Z" suffixes. X X---------------- XREPOST Handling: X---------------- XWarning: X Repost handling is not a configurable parameter within the X rkive configuration file at this time. X XADD_REPOST_SUFFIX define added. X This define allows the administrator to configure the software to X add "-repost" (or whatever is defined in REPOST_SUFFIX) to the X end of all files that are marked as REPOST by the newsgroup moderator. X The suffix is added prior to compression. This feature should only be X configured/exist on systems whose filename limits are greater than 14. X XMV_ORIGINAL define added. X This define allows the administrator to configure the software to X move the original article into a "originals" directory in the X problems directory. The inbound reposted article is placed into X the archive in the correct position. X XIf neither define is specified then the inbound article is placed into Xthe archive in the correct position only if the initial article is not Xin the archive. Otherwise the reposted article is placed in the problems Xdirectory as normal duplicate articles are now. X X----------------- XPATCHES Handling: X----------------- Xrkive supports the new Auxiliary header "Patch-To:". The Patch-To: line Xwill exist for articles that are patches to previously posted software. XThe Patch-To: line only appears in articles that are posted, "Official", Xpatches. The initial postings would not contain the Patch-To: auxiliary Xheader line. X XAuxiliary Headers For Patch Postings: X X Submitted-by: Kent Landfield X Posting-number: Volume 23, Issue 14 X-> Patch-To: Volume 22, Issue 122 X Archive-name: rkive/patch1 X XThere are two different types of handling with regards to patches. X X Package - This type of archiving of patches places the patches X in the same directory that the initial source was X posted to. This type of archiving is only available X to newsgroup archives that are using Archive-Name X archiving. X X Historical - This type of archiving patches is done by sites that X want to place the the patches in the volume/issue in X which the patch originally arrived. X XArchive recognizes that the Patch-To: line indicates the article is Xa patch. For Archive-Name archiving which has specified "Package" Xpatches archiving in the configuration file, rkive puts the article Xinto the directory that contained the initial posting (volume22/rkive). XFor Archive-Name that has not specified Package archiving or for XVolume/Issue archiving, the article would still be labeled as Xvolume23/rkive/patch01 or volume23/v23i014 respectively. X Xrkive also writes a .patchlog file in the BASEDIR for the newsgroup Xthat is used to track patches to originally posted software. The X.patchlog is going to be used for the "random software downloader :-)" Xso that complete software packages (sources and patches) can be requested Xfrom sites that do not use combined Archive-Name and Package archiving. XThe format of the .patchlog file is: X# X# Patchlog for comp.sources.whoknows X# X# Path To Initial Initial Current Current X# Patchfile Volume Issue Volume Issue X# Xbb/patch01 22 105 23 77 X or if volume issue format.. Xv47i022 22 105 23 77 X X------------------------- XArticle Header Reduction: X------------------------- XArticles that are stored just as they arrived on your system are potentially Xwasting disk space. Certain rfc822/rfc1036 header lines are of little use Xafter the article is archived. If you wish to have the headers "trimmed" Xwhen the file is archived, assure that REDUCE_HEADERS is defined. Currently Xall header lines that are *not* either; X X From:, Newsgroups:, Subject:, Message-ID:, and Date: X Xwill be removed. This can produce a savings of as much as 200 to 500 Xbytes per archived article. X XSee news_arc.c if you wish to add or subtract header lines to be kept. XThe modifications need to be made to the hdrstokeep table just above the Xkeep_line() function. X X--------- XSecurity: X--------- Xrkive sets the ownership, group and modes on the archived members according Xto the information specified in the configuration file. Currently though, Xrkive uses the default umask for creating the log and index files. X Xrkive will not archive files outside of the BASEDIR specified in the Xconfiguration file so a "prankster" can not do nasty things to your Xsystem files by having an Archive-name line like: X Archive-name: ../../../../../../etc/passwd X XIt will also not overwrite duplicate files. They are stored underneath Xthe problems directory specified in the configuration file. The admin Xis alerted to the fact and it then becomes a manual cleanup problem. X X ------- X article X ------- X XArticle allows you to view the article headers in much the same manner Xthat you use a printf statement. This was initially done for debugging Xpurposes but I quickly found that it was extremely useful in dealing Xwith news articles in general. It works great in shell scripts to view Xarticles that need to be read.... Also super for perusing the archives Xdirectly and generating indexes to the archives in *many* different Xways...:-) X X -------- X ckconfig X -------- X XThis program is used by the admin to verify just how rkive will Xinterpret the variable specifications in an archive configuration Xfile. If you have problems, it will bomb out when it encounters Xthe problem. Not real smart but it does the job.. X X------------------------------------------------------------------------ XThis software set was developed under an archiving model similar to Xthat maintained currently on uunet. It was intended that the archiving Xfacilities were more of a "site" facility and not an individuals Xfacility. (That is unless the individual owned the site :-)). I have Xnot tried to use rkive for maintaining a private (many on a single machine) Xarchive. There does not seem to be any reason why it would not work. It Xjust hasn't been done. rkive will accept an rkive.cf file specified on the Xcommand line so it would be possible for an individual to have their own Xmini archive directory structure. This is not recommended if the site is Xdoing archiving since the software will store multiple copies thus wasting Xmore disk space than it is worth. Aside from that, if someone does try it, Xlet me know how it turns out. :-) :-) X------------------------------------------------------------------------ XCredits: X-------- XI have to give credit to where credit is do. X XI used the code in header.c of the News 2.11 as the basis of ideas for Xdealing with the article headers. The code I have written is not the same Xbut most of the concepts and some of the flow control resulted from reviewing Xhow it was "suppose to be done". (rfcs only go so far.. :-)) For that I Xthank rick adams and the authors of news for the excellent code to study Xfrom.. :-) X XI would also like to thank my beta testers for the headaches of dealing Xwith me, with forcing different ideas on me at a time when I was "almost" Xwilling to listen :-) and for the many different "full redistribution of Xsources" everytime I had a new version. Specifically I want to thank Xeric@amperif (Eric Johnson) and denny@mcmi (Dennis Page) for putting up Xwith me.. :-) X------------------------------------------------------------------------ X XPlease read all the directions below before you proceed any Xfurther, and then follow them carefully. X X -------------- X Installation X -------------- X XThis package uses Doug Gwyn's directory access routines posted in Xcomp.sources.unix/volume9 (with the bug fix as well). You may need Xto get a copy if you don't already have one and your system does Xnot support POSIX Compatible directory access routines. X X1) Take the time to format and read the man pages prior to continuing. X make man | less/pg/more X X2) Review/modify rkive.h to make sure system defines are correct. X X3) Determine the method for directory creation and edit the Makefile X accordingly. X X4) make X X This will attempt to make the software in the current directory. X X5) Put rkive, ckconfig, and article into a public directory X (normally /usr/local/bin), and put a template of the rkive X configuration file (if one does not exist) into a library directory X (normally as /usr/local/lib/rkive.cf). Place the man pages in the X appropriate man directories for your site. X X X6) I have set up an account for the source archives. This is not really X necessary but is a personal preference. Archive needs to be run as X root *if* you do not have the mkdir () and wish to use the builtin X since it needs to use mknod() to create directories. X X ---x--x--x 1 root archive 43048 Apr 9 16:38 /usr/local/bin/rkive X ---x--x--x 1 src archive 14836 Apr 9 16:38 /usr/local/bin/article X ---x--x--x 1 src archive 27448 Apr 9 16:38 /usr/local/bin/ckconfig X -r--r--r-- 1 src archive 6173 Apr 9 16:40 /usr/local/lib/rkive.cf X X7) Re-read the manual entry for rkive.1 and rkive.5. X X8) Modify the template rkive configuration file to reflect the local X archive conditions. ckconfig should be used in order to check the X information that you have just entered/modified in the rkive.cf file. X X9) VERY IMPORTANT! If you have a problem, there's someone else out there X who either has had or will have the same problem. Please send all X patches, ideas, etc to kent@ssbell (or uunet!ssbell!kent) so that I X can continue to improve the functionality and portability of this X package. END_OF_FILE if test 14704 -ne `wc -c <'README'`; then echo shar: \"'README'\" unpacked with wrong size! fi # end of 'README' fi if test -f 'article.h' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'article.h'\" else echo shar: Extracting \"'article.h'\" \(4444 characters\) sed "s/^X//" >'article.h' <<'END_OF_FILE' X/* X** @(#)article.h 1.1 6/1/89 X** X*/ X X#define BUFLEN 128 /* standard buffer size */ X#define LBUFLEN 1024 /* big buffer size */ X#define PATHLEN 512 /* length of longest source string */ X#define DATELEN 64 /* length of longest allowed date string */ X#define TRUE 1 X#define FALSE 0 X X/* X** Format output called from defines X*/ X#define ARCHIVE 1 X#define ARTICLE 2 X X/* X** Line type defines - index uses X*/ X X#define FROM 1 X#define PATH 2 X#define NEWSGROUP 3 X#define SUBJECT 4 X#define MSG_ID 5 X#define REPLY_TO 6 X#define REFERENCES 7 X#define DATE 8 X#define EXPIRE 9 X#define CONTROL 10 X#define SENDER 11 X#define FOLLOWUP_TO 12 X#define DISTRIBUTION 13 X#define ORGANIZATION 14 X#define NUMLINES 15 X#define KEYWORDS 16 X#define SUMMARY 17 X#define APPROVED 18 X#define SUPERSEDES 19 X#define XREF 20 X#define POSTING_NUMBER 21 X#define SUBMITTED_BY 22 X#define ARCH_NAME 23 X#define ARTICLEID 24 X#define PATCH_TO 25 X#define OTHER 99 X X/* X** article header storage structure X*/ X Xstruct header { X char from[BUFLEN]; /* From: */ X char path[PATHLEN]; /* Path: */ X char nbuf[LBUFLEN]; /* Newsgroups: */ X char subject[BUFLEN]; /* Subject: */ X char ident[BUFLEN]; /* Message-ID: */ X char replyto[BUFLEN]; /* Reply-To: */ X char references[BUFLEN]; /* References: */ X char subdate[DATELEN]; /* Date: (submission) */ X time_t subtime; /* subdate in secs */ X char expdate[DATELEN]; /* Expires: */ X char ctlmsg[PATHLEN]; /* Control: */ X char sender[BUFLEN]; /* Sender: */ X char followup_to[BUFLEN]; /* Followup-to: */ X char distribution[BUFLEN]; /* Distribution: */ X char organization[BUFLEN]; /* Organization: */ X char numlines[8]; /* Lines: */ X int intnumlines; /* Integer version */ X char keywords[BUFLEN]; /* Keywords: */ X char summary[BUFLEN]; /* Summary: */ X char approved[BUFLEN]; /* Approved: */ X char xref[BUFLEN]; /* Xref: */ X char supersedes[BUFLEN]; /* Supersedes: */ X char submitted_by[BUFLEN]; /* Submitted_by: */ X char posting_num[BUFLEN]; /* Posting-number: */ X char archive_name[BUFLEN]; /* Archive-name: */ X char patch_to[BUFLEN]; /* Patch-To: */ X}; X X/* X** Type of archive file defines X*/ X#define NORMAL 0 X#define INFORMATIONAL 1 X#define PATCH 2 X X/* X** archive information structure X*/ X Xstruct archive_rec { X char newsgroup[LBUFLEN]; /* news group */ X char newsarticle[LBUFLEN]; /* news article */ X char filename[BUFLEN]; /* destination file */ X int volume; /* storage volume */ X int issue; /* article issue number */ X int rectype; /* type of article */ X /* NORMAL article = 0 */ X /* INFORMATIONAL = 1 */ X /* PATCH = 2 */ X int repost; /* REPOST'ed article ? */ X int patch_volume; /* Initially posted */ X /* storage volume */ X int patch_issue; /* Initially posted */ X /* article issue number*/ X char description[BUFLEN]; /* information */ X char author_name[BUFLEN]; /* author full name */ X char author_signon[BUFLEN]; /* author sign on */ X}; X Xextern FILE *logfp; X X#ifndef ARTICLE_DEF X#define ARTICLE_DEF 1 X struct archive_rec article; X struct header header; X char s[BUFSIZ]; X int debug; X int verbose; X#else X extern char s[]; X extern int debug; X extern int verbose; X extern struct archive_rec article; X extern struct header header; X#endif /* ARTICLE_DEF */ END_OF_FILE if test 4444 -ne `wc -c <'article.h'`; then echo shar: \"'article.h'\" unpacked with wrong size! fi # end of 'article.h' fi if test -f 'cfg.h' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'cfg.h'\" else echo shar: Extracting \"'cfg.h'\" \(709 characters\) sed "s/^X//" >'cfg.h' <<'END_OF_FILE' X/* X** @(#)cfg.h 1.1 6/1/89 X*/ X#include "rkive.h" X X#ifndef CFG X#define CFG 1 X int num; X char *progname; X struct group_archive group[NUM_NEWSGROUPS]; X FILE *errfp; X FILE *logfp; X int fill_in_defaults; X extern char *config_file; X extern char problems_dir[]; X extern char spooldir[]; X extern char compress[]; X extern char log[]; X extern char log_format[]; X extern char index[]; X extern char index_format[]; X extern char mail[]; X extern int default_type; X extern int default_patch_type; X extern int default_owner; X extern int default_group; X extern int default_modes; X#else X extern int num; X extern char *progname; X extern struct group_archive group[]; X extern FILE *errfp; X extern FILE *logfp; X#endif /* CFG */ END_OF_FILE if test 709 -ne `wc -c <'cfg.h'`; then echo shar: \"'cfg.h'\" unpacked with wrong size! fi # end of 'cfg.h' fi if test -f 'ckconfig.1' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'ckconfig.1'\" else echo shar: Extracting \"'ckconfig.1'\" \(3110 characters\) sed "s/^X//" >'ckconfig.1' <<'END_OF_FILE' X'br "@(#)ckconfig.1 1.1 6/1/89" X.TH CKCONFIG 1 X.SH NAME Xckconfig \- check the rkive configuration file setup X.SH SYNOPSIS X.B ckconfig X[ -V ] [ -f config_file ] X.SH DESCRIPTION X.I ckconfig Xis used to verify a new or modified USENET archiver configuration file. X.PP XThe output is displayed as is shown in the sample below. X.nf X X Configuration Check for /usr/local/lib/rkive.cf X X Global Defines X XNews Directory: /usr/spool/news XProblems Directory: /usenet/problems XGlobal Logfile: /usenet/archive.log XGlobal Index: NO INDEXING XLogfile Format: %B %a\t%T XIndex Format: NOT SPECIFIED XMail Results To: kent,rick X XThe following values are used if the administrator has Xnot set a value for a necessary configuration item(s). X XArchive Type: Volume-Issue XPatches Type: Historical XDefault Owner: 10 XDefault Group: 11 XDefault Modes: 444 XCompression: NO COMPRESSION X X Newsgroup Archive Configuration X Xcomp.sources.unix newsgroup archived to /usenet/unix X Archive Type: Archive-Name X Patches Type: Package X Owner: 10 X Group: 11 X File Modes: 444 X Logfile: /usenet/unix/log X Index File: /usenet/unix/index X Logfile Format: NOT SPECIFIED (DEFAULT) X Index Format: %B %a %T X Mail Results: NO ONE (DEFAULT) X Compression: NO COMPRESSION (*NO* DEFAULT) X X Xcomp.sources.x newsgroup archived to /usenet/x X Archive Type: Volume-Issue X Patches Type: Historical X Owner: 10 X Group: 11 X File Modes: 444 X Logfile: /usenet/x/log X Index File: /usenet/x/index X Logfile Format: NOT SPECIFIED (DEFAULT) X Index Format: %B %a %T X Mail Results: mark, chris X Compression: /usr/local/bin/compress X X.nr X X.PP XThe "(DEFAULT)" indicates that the global value will be used since Xthe administrator did not specify a value specific to that newsgroup. XThe "(*NO* DEFAULT)" indicates that there is no global value to Xbe substituted. In the entry for comp.sources.unix displayed above, Xthere was no compression routine specified in the newsgroups entry Xand there is no globally defined compression routine. X.SH OPTIONS X.IP "-V" XPrint the version number and patchlevel of the current executable. X.IP "-f config_file" XExecute ckconfig using the specified config_file as the input file to Xbe checked. X.IP "-g" XThis option allows the global default values to be displayed in the Xnewsgroup variables when there is nothing specified in the newsgroup Xentry of the configuration file. If the above example had been run Xusing this option, the output would have been: X.nf X Xcomp.sources.unix newsgroup archived to /usenet/unix X Archive Type: Archive-Name X Patches Type: Package X Owner: 10 X Group: 11 X File Modes: 444 X Logfile: /usenet/unix/log X Index File: /usenet/unix/index X Logfile Format: %B %a\t%T X Index Format: %B %a %T X Mail Results: kent,rick X Compression: NO COMPRESSION (*NO* DEFAULT) X X.nr X.LP X.SH FILES X/usr/local/lib/rkive.cf X.SH "SEE ALSO" Xarticle(1), rkive(1), rkive(5) X.SH BUGS XNone known. END_OF_FILE if test 3110 -ne `wc -c <'ckconfig.1'`; then echo shar: \"'ckconfig.1'\" unpacked with wrong size! fi # end of 'ckconfig.1' fi if test -f 'disp_grp.c' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'disp_grp.c'\" else echo shar: Extracting \"'disp_grp.c'\" \(3810 characters\) sed "s/^X//" >'disp_grp.c' <<'END_OF_FILE' X/* X** X** This software is Copyright (c) 1989 by Kent Landfield. X** X** Permission is hereby granted to copy, distribute or otherwise X** use any part of this package as long as you do not try to make X** money from it or pretend that you wrote it. This copyright X** notice must be maintained in any copy made. X** X** X** History: X** Creation: Tue Feb 21 08:52:35 CST 1989 due to necessity. X** X*/ X#ifndef lint Xstatic char SID[] = "@(#)disp_grp.c 1.1 6/1/89"; X#endif X X#include X#include X#include X#include X#include X#include "cfg.h" X Xextern fill_in_defaults; Xextern FILE *logfp; X Xstruct passwd *pw; Xstruct passwd *getpwuid(); Xstruct group *gr; Xstruct group *getgrgid(); X Xchar *what_to_print(ng_val, gbl_val, str) Xchar *ng_val, *gbl_val, *str; X{ X static char rtbuf[50]; X X /* X ** If the newsgroup variable has been X ** specified, return that value. X */ X if (*(ng_val)) X return (ng_val); X X /* X ** If no global value has been specified X ** so that there is no default, build the X ** display string and return. X */ X if (!*gbl_val) { X (void) sprintf(rtbuf,"%-15s (*NO* DEFAULT)", str); X return (rtbuf); X } X X /* X ** Ok, here we have a global value. X ** Check if the user has requested to display global default X ** values in variables that are not specifically assigned X ** for the newsgroup... X */ X if (fill_in_defaults) X return (gbl_val); X X (void) sprintf(rtbuf,"%-15s (DEFAULT)", str); X return (rtbuf); X} X X Xdisplay_group_info(ng) Xstruct group_archive *ng; X{ X if (!*ng->location) X (void) fprintf(logfp,"\007\007%s does not have an archived location\n", X ng->ng_name); X else X (void) fprintf(logfp,"%s newsgroup archived to %s\n", X ng->ng_name, ng->location); X X (void) fprintf(logfp,"\tArchive Type: %s\n", X ng->type == ARCHIVE_NAME ? "Archive-Name" : X ng->type == VOLUME_ISSUE ? "Volume-Issue" : X "Article-Number"); X (void) fprintf(logfp,"\tPatches Type: %s\n", X ng->patch_type == PACKAGE ? "Package" : "Historical"); X X /* X ** The getpwuid() and getgrgid() calls have been previously made X ** thus verifying the entries exist. No real reason to check if X ** calls fail. If they do, something a lot bigger than this program X ** is hosed... X */ X pw = getpwuid(ng->owner); X (void) fprintf(logfp,"\tOwner: %d <%s>\n", X ng->owner, pw->pw_name); X gr = getgrgid(ng->group); X (void) fprintf(logfp,"\tGroup: %d <%s>\n", X ng->group, gr->gr_name); X X (void) fprintf(logfp,"\tFile Modes: %o\n", ng->modes); X (void) fprintf(logfp,"\tLogfile: %s\n", X what_to_print(ng->logfile, log, "NO LOGGING")); X (void) fprintf(logfp,"\tIndex File: %s\n", X what_to_print(ng->index, index, "NO INDEXING")); X (void) fprintf(logfp,"\tLogfile Format: %s\n", X what_to_print(ng->logformat, log_format, "NOT SPECIFIED")); X (void) fprintf(logfp,"\tIndex Format: %s\n", X what_to_print(ng->indformat, index_format, "NOT SPECIFIED")); X (void) fprintf(logfp,"\tMail Results: %s\n", X what_to_print(ng->mail_list, mail, "NO ONE")); X (void) fprintf(logfp,"\tCompression: %s\n", X what_to_print(ng->compress, compress, "NO COMPRESSION")); X X if ((ng->patch_type == PACKAGE) && (ng->type != ARCHIVE_NAME)) { X (void) fprintf(logfp,"\nWARNING: Package Patches archiving is only\n"); X (void) fprintf(logfp," used with Archive-Name archiving.\n"); X } X} X END_OF_FILE if test 3810 -ne `wc -c <'disp_grp.c'`; then echo shar: \"'disp_grp.c'\" unpacked with wrong size! fi # end of 'disp_grp.c' fi if test -f 'efopen.c' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'efopen.c'\" else echo shar: Extracting \"'efopen.c'\" \(828 characters\) sed "s/^X//" >'efopen.c' <<'END_OF_FILE' X/* X** X** This software is Copyright (c) 1989 by Kent Landfield. X** X** Permission is hereby granted to copy, distribute or otherwise X** use any part of this package as long as you do not try to make X** money from it or pretend that you wrote it. This copyright X** notice must be maintained in any copy made. X** X** X** History: X** Creation: Tue Feb 21 08:52:35 CST 1989 due to necessity. X** X*/ X#ifndef lint Xstatic char SID[] = "@(#)efopen.c 1.1 6/1/89"; X#endif X X#include X Xextern FILE *errfp; X XFILE *efopen(file,mode) Xchar *file, *mode; X{ X FILE *fp; X FILE *fopen(); X void exit(); X X if ((fp = fopen (file, mode)) == NULL) { X (void) fprintf (errfp, "Can't open file %s\n", file); X exit(1); X } X return (fp); X} END_OF_FILE if test 828 -ne `wc -c <'efopen.c'`; then echo shar: \"'efopen.c'\" unpacked with wrong size! fi # end of 'efopen.c' fi if test -f 'makedir.c' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'makedir.c'\" else echo shar: Extracting \"'makedir.c'\" \(3647 characters\) sed "s/^X//" >'makedir.c' <<'END_OF_FILE' X/* X** X** This software is Copyright (c) 1989 by Kent Landfield. X** X** Permission is hereby granted to copy, distribute or otherwise X** use any part of this package as long as you do not try to make X** money from it or pretend that you wrote it. This copyright X** notice must be maintained in any copy made. X** X** X** History: X** Creation: Tue Feb 21 08:52:35 CST 1989 due to necessity. X** X*/ X#ifndef lint Xstatic char SID[] = "@(#)makedir.c 1.1 6/1/89"; X#endif X X#include X#include X#include X#include X#include "rkive.h" X Xextern char *progname; X Xextern int verbose; Xextern int test; X Xextern FILE *errfp; Xextern FILE *logfp; X Xint makedir(dirpath, mode, owner_id, group_id) X char *dirpath; X int mode; X int owner_id; X int group_id; X{ X#ifndef HAVE_MKDIR X# ifndef USE_SYSMKDIR X X char *strcat(); X char *strncpy(); X char *strcpy(); X X register i; X register slen = 0; X X char parent[MAXNAMLEN]; X X#endif /* USE_SYSMKDIR */ X X char crnt_dir[MAXNAMLEN]; X X#endif /* HAVE_MKDIR */ X X if ((strlen(dirpath) == 0) || (dirpath[0] == NULL)) { X (void) fprintf(errfp,"%s: cannot make %s\n", progname,dirpath); X return(-1); X } X X if (verbose) { X (void) fprintf(logfp,"making\t<%s>\n",dirpath); X if (test) X return(0); X } X X#ifdef HAVE_MKDIR X X /* X ** mkdir function supplied in system library. X */ X X if (mkdir(dirpath,mode) != 0) X return(-1); X X (void) chown(dirpath, owner_id, group_id); X X#else X# ifdef USE_SYSMKDIR X X /* X ** Use the system mkdir executable.. why? X */ X X /* build up command */ X X (void) sprintf(crnt_dir, "/bin/mkdir %s", dirpath); X X if (system(crnt_dir) != 0) X return(-1); X X (void) chown(dirpath, owner_id, group_id); X (void) chmod(dirpath, mode); X X# else X X /* X ** Builtin mkdir function in use ... X ** X ** Save the parent path X */ X X i = 0; X parent[0] = '\0'; X X while (dirpath[i]) { X if (dirpath[i] == '/') X slen = i + 1; X ++i; X } X X if (slen) /* Is there a parent string to save ? */ X (void) strncpy(parent, dirpath, slen); X X (void) strcpy(parent+slen, "."); X X /* Does the parent directory exist and is it writable ? */ X X if (access(parent, 02)) { X (void) fprintf(errfp,"%s: cannot access %s\n", progname,parent); X return(-1); X } X X /* make the actual directory file */ X X if ((mknod(dirpath, S_IFDIR | mode, 0)) < 0) { X (void)fprintf(errfp,"%s: can't make directory %s\n",progname,dirpath); X return(-1); X } X X (void) chown(dirpath, owner_id, group_id); X X /* need to create the "." directory entry */ X X (void) strcpy(crnt_dir, dirpath); X (void) strcat(crnt_dir, "/."); X X if ((link(dirpath, crnt_dir)) < 0) { X /* X ** Could not link the directory to it's "." entry. X ** Clean up and return. X */ X (void) unlink(dirpath); X (void) fprintf(errfp, "%s: cannot link [%s]\n", progname,crnt_dir); X return(-1); X } X X /* need to create the "." directory entry */ X X (void) strcat(crnt_dir, "."); X X if ((link(parent, crnt_dir)) < 0) { X /* X ** Could not link the parent directory to the ".." entry. X ** Clean up and return. X */ X crnt_dir[strlen(crnt_dir)] = '\0'; X (void) unlink(crnt_dir); X (void) unlink(dirpath); X (void) fprintf(errfp, "%s: cannot link [%s]\n",progname,crnt_dir); X return(-1); X } X#endif /* USE_SYSMKDIR */ X#endif /* HAVE_MKDIR */ X return(0); X} END_OF_FILE if test 3647 -ne `wc -c <'makedir.c'`; then echo shar: \"'makedir.c'\" unpacked with wrong size! fi # end of 'makedir.c' fi if test -f 'patchlevel.h' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'patchlevel.h'\" else echo shar: Extracting \"'patchlevel.h'\" \(61 characters\) sed "s/^X//" >'patchlevel.h' <<'END_OF_FILE' X/* X** @(#)patchlevel.h 1.1 6/1/89 X*/ X#define PATCHLEVEL 0 END_OF_FILE if test 61 -ne `wc -c <'patchlevel.h'`; then echo shar: \"'patchlevel.h'\" unpacked with wrong size! fi # end of 'patchlevel.h' fi if test -f 'rename.c' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'rename.c'\" else echo shar: Extracting \"'rename.c'\" \(1055 characters\) sed "s/^X//" >'rename.c' <<'END_OF_FILE' X/* X** X** This software is Copyright (c) 1989 by Kent Landfield. X** X** Permission is hereby granted to copy, distribute or otherwise X** use any part of this package as long as you do not try to make X** money from it or pretend that you wrote it. This copyright X** notice must be maintained in any copy made. X** X** X** History: X** Creation: Tue Feb 21 08:52:35 CST 1989 due to necessity. X** X*/ X#ifndef lint Xstatic char SID[] = "@(#)rename.c 1.1 6/1/89"; X#endif X X#include X Xextern FILE *errfp; X X/* X** rename(sfn, dfn) X** rename char *sfn to char *dfn X*/ X Xint rename(sfn, dfn) Xchar *sfn; /* source file name */ Xchar *dfn; /* destination file name */ X{ X int rcd; X X (void) unlink(dfn); /* remove the destination file */ X X if ((rcd = link(sfn, dfn)) == -1) X (void) fprintf(errfp, "Can't link %s to %s\n", sfn, dfn); X else if ((rcd = unlink(sfn)) == -1) X (void) fprintf(errfp, "Can't unlink %s\n", sfn); X return(rcd); X} X END_OF_FILE if test 1055 -ne `wc -c <'rename.c'`; then echo shar: \"'rename.c'\" unpacked with wrong size! fi # end of 'rename.c' fi if test -f 'rkive.1' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'rkive.1'\" else echo shar: Extracting \"'rkive.1'\" \(4927 characters\) sed "s/^X//" >'rkive.1' <<'END_OF_FILE' X'br "@(#)rkive.1 1.1 6/1/89" X.TH RKIVE 1 X.SH NAME Xrkive \- archive USENET source groups X.SH SYNOPSIS X.B rkive X[ -dstuvV ] [ -n newsgroup ] X.SH DESCRIPTION X.I rkive Xis used to archive the USENET sources groups to an alternate Xlocation as specified in an rkive configuration file. Archives can Xbe maintained in one of three ways: X.PP X.I Archive-Name - XThe moderators of most sources groups assign an official Archive-Name Xto each article that gets submitted to the net. In this manner, each file Xhas a "new-login" or "elm/part06" type of format. For multi-part postings, Xa subdirectory is created (as indicated in the elm example) to hold the Xseparate "parts". This format is used by many large archive sites because Xit is easier for retrieval via mail request software such as netlib and Xthe filenames give hints as to what the software is. X.PP X.I Volume-Issue - XSoftware sent via most moderated groups have an assigned Volume and Issue Xnumber. This allows the moderators to track and reference the individual Xitems that have been posted to the group. Each individual article is given Xan "Issue" number. The Issues are grouped together into a "Volume". There Xare roughly 100 articles in each Volume but this is an arbitrary split Xtotally up to the moderator. This format is extremely useful when the Xsoftware archives are cataloged. It makes searching of the files quicker Xand verification of complete volumes easier. This archive format is Xrecommended for any site that will be doing massive searches of the Xindividual volumes since it keeps the quadratic nature of directory searches Xfrom making your life miserable. X.PP X.I Article Number - XThe news software stores the articles locally by naming the news article Xby a number generated on every site. The Article Number ordering is unique Xto each site. If an Article Number archive is requested (or required by the Xnewsgroup), the news article file is copied to the directory specified in Xthe archive configuration file. The name of the archived article will match Xthe original name generated by the news software. X.PP XBy means of a configuration file (see rkive(5)), the archive administrator Xis able to control how archiving is performed. The administrator can specify Xon a per newsgroup basis: X.nf X X o The type of the archiving, such as Volume-Issue X Archive-Name, or Article Number archiving, X o Where the newsgroup archive is to be stored on disk, X o The location of log file for the newsgroup, X o The format of the logfile records, X o The location of index file for the newsgroup, X o The format of the index file records, X o A list of users to be sent mail when an article is archived, X o The owner/group and modes of each archived member, and X o Whether the archived members should be compressed or not. X X.nr X.PP XThis program is normally started from cron on a daily basis. Archiving Xof newly arrived software is only done once for each file. If rkive Xis started daily, the posted sources are placed into the sources archive Xthe first day the software arrives on the machine instead of having to Xwait for expire -a to run. X.PP X.SH OPTIONS X.IP "-d" 6 XTurn on debugging output. This is more "verbose" than the verbose Xoption below. (-v assumed) X.IP "-f config_file" XThis option allows the user to specify the file to be used as the Xrkive configuration file. X.IP "-g" XThis option allows the global default values to be displayed in the Xnewsgroup variables when there is nothing specified in the newsgroup Xentry of the configuration file. Only applicable with debugging and Xif a verbose status is requested. X.IP "-n newsgroup" XThis restricts the execution to just the newsgroup indicated. The newsgroup Xneeds to be specified in a "comp.sources.unix" format. Only one newsgroup Xcan be processed in this manner. If this option is not specified, then all Xnewsgroups specified in the rkive configuration file are processed. X.IP "-s" XPrint a quick status of the sources that currently reside Xwithin the USENET newsgroup directories for newsgroups Xspecified in the rkive configuration file. The status Xdisplays whether an article has been archived or if it is Xstill waiting to be archived. If it has been archived, the Xarchived location is displayed. X.IP "-t" XTest only, no archiving is done. Print a report of all Xactions that would take place if it was called without the Xtest flag. (-v assumed) X.IP "-u" XUnconditionally archive the articles. Normally the software Xwill not overwrite an archive member if a file exists with Xthe same name as the destination archived article. The use Xof this option ignores the check and overwrites an existing Xarchive member. X.IP "-v" XVerbose. Prints a file by file account of the actions. X.IP "-V" XPrint the version number and patchlevel of the current executable. X.LP X.SH FILES X/usr/local/lib/rkive.cf X.archived X.patchlog X.SH "SEE ALSO" Xarticle(1), rkive.cf(5), ckconfig(1) X.SH BUGS XNone known. END_OF_FILE if test 4927 -ne `wc -c <'rkive.1'`; then echo shar: \"'rkive.1'\" unpacked with wrong size! fi # end of 'rkive.1' fi if test -f 'str.c' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'str.c'\" else echo shar: Extracting \"'str.c'\" \(2587 characters\) sed "s/^X//" >'str.c' <<'END_OF_FILE' X/* X** X** This software is Copyright (c) 1989 by Kent Landfield. X** X** Permission is hereby granted to copy, distribute or otherwise X** use any part of this package as long as you do not try to make X** money from it or pretend that you wrote it. This copyright X** notice must be maintained in any copy made. X** X** X** History: X** Creation: Tue Feb 21 08:52:35 CST 1989 due to necessity. X** X*/ X#ifndef lint Xstatic char SID[] = "@(#)str.c 1.1 6/1/89"; X#endif X X/* X** Strchr() and Strrchr() are included for portability sake only. X*/ X X#ifndef NULL X#define NULL 0 X#endif X X#ifndef lint X/* X** strchr: X** X** strchr(str, c) returns a pointer to the first place in X** str where c occurs, or NULL if c does not occur in str. X** X** char *strchr(str, c) char *str, c; { return (str); } X** X*/ X Xchar *strchr(str, c) Xregister char *str, c; X{ X for (;;) { X if (*str == c) X return (str); X if (!*str++) X return (NULL); X } X} X X/* X** strrchr: X** X** strrchr(str, c) returns a pointer to the last place in X** str where c occurs, or NULL if c does not occur in str. X** X** char *strrchr(str, c) char *str, c; { return (str); } X*/ X Xchar *strrchr(str, c) Xregister char *str, c; X{ X register char *t; X X t = NULL; X do { X if (*str == c) X t = str; X } while (*str++); X return(t); X} X#endif /* lint */ X X/* X** strstrip: X** X** strstrip(str) returns a pointer to the first non-blank character X** in str. It strips all blanks, and tabs from the front and back X** of a string as well as strip off newlines from the rear. X** X** char *strstrip(str) char *str; { return (str); } X*/ X Xchar *strstrip(str) Xregister char *str; X{ X register char *cp, *dp; X X cp = str; X dp = ((str+strlen(str))-1); X X while(*cp && (*cp == ' ' || *cp == '\t')) X cp++; X X while(dp != cp && (*dp == ' ' || *dp == '\t' || *dp == '\n')) X --dp; X *(dp+1) = '\0'; X X return(cp); X} X X/* X** substr: X** X** substr(str, substr) returns a pointer to the first place in X** str where the substring substr occurs, or NULL if substr does X** not occur in str. X** X** char *substr(str, substr) char *str, *substr; { return (str); } X*/ Xchar *substr(from, find) X char *from, *find; X{ X register char *sp, *np; X register int len; X char *strchr(); X X np = from; X len = strlen(find); X X while ((sp = strchr(np,*find)) != NULL) { X if (strlen(sp) < len) X break; X if (strncmp(sp,find,len) == 0) X return(sp); X np = sp + 1; X } X return(NULL); X} END_OF_FILE if test 2587 -ne `wc -c <'str.c'`; then echo shar: \"'str.c'\" unpacked with wrong size! fi # end of 'str.c' fi if test -f 't.cf' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'t.cf'\" else echo shar: Extracting \"'t.cf'\" \(957 characters\) sed "s/^X//" >'t.cf' <<'END_OF_FILE' X# X# @(#)t.cf 1.1 6/1/89 X# X# Test rkive configuration file. X# X###################################################################### X# GLOBAL PARAMETERS # X###################################################################### XSPOOLDIR=/usr/spool/news XPROBLEMS=/usenet/problems XTYPE= Volume-Issue XPATCHES= Package XMAIL=kent XOWNER=src XGROUP=archive XMODE=0444 X#LOG=/usenet/test.log X#LOG_FORMAT= "%a %T" X#INDEX= /usenet/test.index X#INDEX_FORMAT= "%B %a %T" X#COMPRESS=/usr/lbin/compress X X###################################################################### X# NEWSGROUP PARAMETERS # X###################################################################### X$$comp.sources.test X BASEDIR: /usenet/test X TYPE: Archive-Name X PATCHES: Package X LOG: /usenet/test/log X INDEX: /usenet/test/index X INDEX_FORMAT: "%B %a %T" X# COMPRESS=/usr/lbin/compress END_OF_FILE if test 957 -ne `wc -c <'t.cf'`; then echo shar: \"'t.cf'\" unpacked with wrong size! fi # end of 't.cf' fi if test -f 'version.c' -a "${1}" != "-c" ; then echo shar: Will not clobber existing file \"'version.c'\" else echo shar: Extracting \"'version.c'\" \(712 characters\) sed "s/^X//" >'version.c' <<'END_OF_FILE' X/* X** X** This software is Copyright (c) 1989 by Kent Landfield. X** X** Permission is hereby granted to copy, distribute or otherwise X** use any part of this package as long as you do not try to make X** money from it or pretend that you wrote it. This copyright X** notice must be maintained in any copy made. X** X** X** History: X** Creation: Tue Feb 21 08:52:35 CST 1989 due to necessity. X** X*/ X#ifndef lint Xstatic char SID[] = "@(#)version.c 1.1 6/1/89"; X#endif X X#include X#include "patchlevel.h" X Xextern char sccsid[]; X Xversion() X{ X void exit(); X X (void) fprintf(stderr,"%s\nPatch level: %d\n", sccsid, PATCHLEVEL); X exit(0); X} END_OF_FILE if test 712 -ne `wc -c <'version.c'`; then echo shar: \"'version.c'\" unpacked with wrong size! fi # end of 'version.c' fi echo shar: End of archive 1 \(of 4\). cp /dev/null ark1isdone MISSING="" for I in 1 2 3 4 ; do if test ! -f ark${I}isdone ; then MISSING="${MISSING} ${I}" fi done if test "${MISSING}" = "" ; then echo You have unpacked all 4 archives. rm -f ark[1-9]isdone else echo You still need to unpack the following archives: echo " " ${MISSING} fi ## End of shell archive. exit 0