From:	SMTP%"terry@SPCVXB.SPC.EDU"  7-SEP-1993 09:00:49.86
To:	EVERHART
CC:	
Subj:	XOVER support for V6.1B8 - performance improvement

Message-Id: <9309051035.AA27208@uu7.psi.com>
Date:         Sun, 5 Sep 1993 10:20:14 GMT
Reply-To: "Terry Kennedy, Operations Mgr." <terry@SPCVXB.SPC.EDU>
Sender: ANU-NEWS Discussion <ANU-NEWS@VM1.NoDak.EDU>
Comments:     Warning -- original Sender: tag was news@SPCUNA.SPC.EDU
From: "Terry Kennedy, Operations Mgr." <terry@SPCVXB.SPC.EDU>
Organization: St. Peter's College, US
Subject:      XOVER support for V6.1B8 - performance improvement
To: Multiple recipients of list ANU-NEWS <ANU-NEWS@VM1.NoDak.EDU>

  I've modified the ANU 6.1B8 distribution to try using the XOVER command
when talking to an NNTP server. XOVER is an optional feature of C News (to
be standard in the forthcoming "Cleanup Release") and a standard feature
(or so I'm told) of INN.

  This change greatly improves performance, especially when the user has
lines and/or postmarks configured in their profile or if they have a display
setting of allitems (or use the "show allitems" command during a session).
Lines and postmarks are now no-penalty items. Allitems is vastly improved,
as the following table shows (all times with lines+postmarks+allitems):

Newsgroup   Item    Stock version    XOVER version
Name        Count   time (seconds)   time (seconds)
---------------------------------------------------
junk          113             3.69             1.37
control      1775            63.36            11.73

  Additionally, this change reduces the load on the serving system.

  On to the details: First, this patch does you no good if you're using an
ANU server (at least until someone adds XOVER support to the ANU server). You
need to be running NNTP 1.5.11 with the C News XOVER patches or the current
release of INN. Applying these patches will do no harm if you don't have such
a server - News will try XOVER, get an error, and fall back to XHDR or HEADER
as appropriate.

  Next, this patch should be classified as *experimental* - I've tested it a
bit and will do more extensive testing with users throughout the coming week,
but I can't guarantee it won't cause problems.

  A patch file which changes newssite.h, newssite.c, and newsutility.c is
attached to the end of this message. You need to apply it with the Unix "patch"
program (as found on several sites in the DECUS UUCP distribution). You will
likely get "offset N lines" messages from patch as I've made other changes to
6.1B8. You'll then need to invoke the newsbuild procedure (which will recompile
_everything_, since newssite.h changed, so settle in for a long evening) and
install the resulting news.exe (presumably under another name) for testing.

  The following new messages may be displayed:

>From "show version":
Support for NNTP use of the ANU-NEWS extension for the 'xover' command
(configuration display message)

When opening a group or reading an item outside the current group:
NNTP XOVER error in Subject field.
NNTP XOVER error in From field.
NNTP XOVER error in Date field.
NNTP XOVER error in Message-ID field.
NNTP XOVER error in References field.
NNTP XOVER error in byte count field.
NNTP XOVER error in Lines field.
(corrupted overview data received from the NNTP server)

When opening a group or reading an item outside the current group:
DEBUG: Asking for items via XOVER
DEBUG: Asking for items via XHDR
DEBUG: Asking for items via HEAD
(informational display showing what method was used to read the info)

At an undetermined time:
DEBUG: Asking for Message-ID's via XHDR");
(informational display showing that Message-ID's were requested)

*** newssite.c_pre-xover        Sun Sep  5 03:55:01 1993
--- newssite.c  Sun Sep  5 03:57:27 1993
***************
*** 810,815 ****
--- 810,817 ----
          "Default action to take with newly created newsgroups, 1 - register, 0
 - don't register"},
      {"CONFIRM_NEW_GROUPS", CONFIRM_NEW_GROUPS,
          "Confirmation of newgroup registration default action"},
+     {"NNTP_USE_XOVER", NNTP_USE_XOVER,
+         "Support for NNTP use of the ANU-NEWS extension for the 'xover' comman
d"},
      {"NNTP_USE_XHDR", NNTP_USE_XHDR,
          "Support for NNTP use of the ANU-NEWS extension for the 'xhdr' command
"},
      {"NNTP_USE_XUPDGROUP", NNTP_USE_XUPDGROUP,

*** newssite.h_pre-xover        Sun Sep  5 03:54:55 1993
--- newssite.h  Sun Sep  5 03:56:31 1993
***************
*** 348,353 ****
--- 348,362 ----
   */

  #define CONFIRM_NEW_GROUPS 1
+
+ /*
+  * NNTP_USE_XOVER determines whether the nntp client will use xover commands
+  * against the remote server. For servers which do not support this, the use
+  * of xover will fill the log file with invalid command entries. XOVER is much
+  * faster than XHDR.
+  */
+
+ #define NNTP_USE_XOVER 1

  /*
   * NNTP_USE_XHDR determines whether the nntp client will use xhdr commands

*** newsutility.c_pre-xover     Sun Sep  5 03:55:18 1993
--- newsutility.c       Sun Sep  5 05:12:10 1993
***************
*** 2128,2133 ****
--- 2128,2236 ----
  static GRP_PTR sgap;
  static unsigned int sgi;

+ /*+ V6.1B8 - 4-Sep-1993 - tmk - Use XOVER command to improve performance */
+ static int xoverni(cp1)
+   char *cp1;
+ {
+   char *cp3;
+   int indx, i, j, x;
+   char *xpitem, *xpsubj, *xpfrom, *xpdate, *xpmsid, *xprefs, *xpsize, *xpline;
+
+   chop_str(cp1,'\n');
+   if (sscanf(cp1,"%d",&indx) != 1) return 0;
+
+   if (sgap->grp_count && find_itm_by_num(sgi,indx,&i)) return 0;
+   else i = sgap->grp_count + 1;
+
+   cp3 = cp1;
+
+   xpitem=cp3;
+   xpsubj=strchr(cp3,'\t');
+   if (xpsubj == NULL)
+     err_line("NNTP XOVER error in Subject field.");
+   else {
+     *xpsubj='\0';
+     xpsubj++;
+   }
+   xpfrom=strchr(xpsubj,'\t');
+   if (xpfrom == NULL)
+     err_line("NNTP XOVER error in From field.");
+   else {
+     *xpfrom='\0';
+     xpfrom++;
+   }
+   xpdate=strchr(xpfrom,'\t');
+   if (xpdate == NULL)
+     err_line("NNTP XOVER error in Date field.");
+   else {
+     *xpdate='\0';
+     xpdate++;
+   }
+   xpmsid=strchr(xpdate,'\t');
+    if (xpmsid == NULL)
+     err_line("NNTP XOVER error in Message-ID field.");
+   else {
+     *xpmsid='\0';
+     xpmsid++;
+   }
+   xprefs=strchr(xpmsid,'\t');
+   if (xprefs == NULL)
+     err_line("NNTP XOVER error in References field.");
+   else {
+     *xprefs='\0';
+     xprefs++;
+   }
+   xpsize=strchr(xprefs,'\t');
+   if (xpsize == NULL)
+     err_line("NNTP XOVER error in byte count field.");
+   else {
+     *xpsize='\0';
+     xpsize++;
+   }
+   xpline=strchr(xpsize,'\t');
+   if (xpline == NULL)
+     err_line("NNTP XOVER error in Lines field.");
+   else {
+   *xpline='\0';
+   xpline++;
+   }
+
+   /* Note that we don't use the Message-ID or References fields. News will
+    * grab the Message-ID's in the rare case that it needs them. Better to
+    * avoid the overhead here.
+    */
+
+   if (sgap->grp_iasize == sgap->grp_count) {
+     sgap->grp_iasize += 10;
+     sgap->grp_ia = realloc(sgap->grp_ia,(sgap->grp_iasize + 1) * (sizeof newsi
tm));
+     }
+   j = sgap->grp_count + 1;
+   while (j > i) {
+     sgap->grp_ia[j] = sgap->grp_ia[j-1];
+     --j;
+     }
+   newsitm.itm_num = indx;
+   newsitm.itm_grp = sgap->grp_num;
+   *(newsitm.itm_id) = '\0';
+   newsitm.itm_recvdate = 0;
+   newsitm.itm_lines = 0;
+   newsitm.itm_flags = NEWS_M_UNREAD;
+   util_subjcpy(newsitm.itm_title,xpsubj);
+   util_fromcpy(newsitm.itm_from,xpfrom);
+   if (sscanf(xpline,"%d",&x) == 1) {
+     newsitm.itm_lines = x;
+     newsitm.itm_flags |= NEWS_M_LINESVALID;
+   }
+   newsitm.itm_postdate = newsitm.itm_recvdate = parse_usenet_date(xpdate);
+   newsitm.itm_cachedate = 0;
+   newsitm.itm_life = 0;
+   newsitm.itm_cid = 0;
+   sgap->grp_ia[i] = newsitm;
+   ++sgap->grp_count;
+   return 1;
+ }
+ /*- V6.1B8 - 4-Sep-1993 - tmk - Use XOVER command to improve performance */
+
  static int initni(cp1)
    char *cp1;
  {
***************
*** 2230,2282 ****
        if (count) {
          sgap = gap;
          sgi = g;
!         sprintf(err_oline,"XHDR Subject %d-%d",first,last);
!         if ((NNTP_USE_XHDR) &&
!             (nntp_get_info(nntp_node,nntp_proto,err_oline,221,0,initni))) {
!           if (profile_display_lines) ds_lines(g);
!           if (profile_display_postmark) ds_postmark(g);
!           if (minfromlen > 0) ds_from(g);
            xhdr_call = 1;
            }
          else {
!           for (indx = first; indx <= last; ++indx) {
!             sprintf(err_oline,"HEAD %d",indx);
!             if (nntp_call(nntp_node,nntp_proto,err_oline,&bufptr) == 221) {
!               newsitm.itm_grp = gap->grp_num;
!               util_subjcpy(newsitm.itm_title,"--Title not retrieved--");
!               newsitm.itm_recvdate = 0;
!               newsitm.itm_lines = 0;
!               newsitm.itm_cachedate = 0;
!               newsitm.itm_life = 0;
!               newsitm.itm_flags = NEWS_M_UNREAD;      /* glass@mgi.com  V6.0-3
  4/2/91 */
!               newsitm.itm_cid = 0;
!             cp1 = chop_str_plus(bufptr,'\n');
!               sscanf(bufptr,"221 %d %s",&newsitm.itm_num,subject);
!               util_idcpy(newsitm.itm_id,subject);
!               while (cp1) {
!               cp2 = chop_str_plus(cp1,'\n');
!               if ((cp3 = chop_str_plus(cp1,':')) != 0) {
!                   while (isspace(*cp3)) ++cp3;
!                   lower_case(cp1);
!                   if (!strcmp(cp1,"subject")) util_subjcpy(newsitm.itm_title,c
p3);
!                   else if (!strcmp(cp1,"from")) util_fromcpy(newsitm.itm_from,
cp3);
!                   else if (!strcmp(cp1,"message-id")) util_idcpy(newsitm.itm_i
d,cp3);
!                   else if (!strcmp(cp1,"lines")) {
!                     if (sscanf(cp3,"%d",&(newsitm.itm_lines)) == 1)
!                       newsitm.itm_flags |= NEWS_M_LINESVALID;
!                     }
!                   else if (!strcmp(cp1,"date"))
!                     newsitm.itm_postdate = newsitm.itm_recvdate = parse_usenet
_date(cp3);
!                 }
!                 cp1 = cp2;
!                 }
!               if (gap->grp_iasize == gap->grp_count) {
!                 gap->grp_iasize += 10;
!                 gap->grp_ia = realloc(gap->grp_ia,(gap->grp_iasize + 1) * (siz
eof newsitm));
                  }
-               gap->grp_ia[++gap->grp_count] = newsitm;
                }
!             }
            }
          }
        if (gap->grp_unread > gap->grp_count) gap->grp_unread = gap->grp_count;
--- 2333,2403 ----
        if (count) {
          sgap = gap;
          sgi = g;
! /*+ V6.1B8 - 4-Sep-1993 - tmk - Use XOVER command to improve performance */
! /*+ V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
!       err_line("DEBUG: Asking for items via XOVER");
! /*- V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
!       sprintf(err_oline,"XOVER %d-%d", first,last);
!         if ((NNTP_USE_XOVER) &&
!             (nntp_get_info(nntp_node,nntp_proto,err_oline,224,0,xoverni))) {
            xhdr_call = 1;
            }
+ /*- V6.1B8 - 4-Sep-1993 - tmk - Use XOVER command to improve performance */
          else {
! /*+ V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
!         err_line("DEBUG: Asking for items via XHDR");
! /*- V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
!         sprintf(err_oline,"XHDR Subject %d-%d",first,last);
!           if ((NNTP_USE_XHDR) &&
!               (nntp_get_info(nntp_node,nntp_proto,err_oline,221,0,initni))) {
!             if (profile_display_lines) ds_lines(g);
!             if (profile_display_postmark) ds_postmark(g);
!             if (minfromlen > 0) ds_from(g);
!             xhdr_call = 1;
!           }
!           else {
! /*+ V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
!           err_line("DEBUG: Asking for items via HEAD");
! /*- V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
!             for (indx = first; indx <= last; ++indx) {
!               sprintf(err_oline,"HEAD %d",indx);
!               if (nntp_call(nntp_node,nntp_proto,err_oline,&bufptr) == 221) {
!                 newsitm.itm_grp = gap->grp_num;
!                 util_subjcpy(newsitm.itm_title,"--Title not retrieved--");
!                 newsitm.itm_recvdate = 0;
!                 newsitm.itm_lines = 0;
!                 newsitm.itm_cachedate = 0;
!                 newsitm.itm_life = 0;
!                 newsitm.itm_flags = NEWS_M_UNREAD;    /* glass@mgi.com  V6.0-3
  4/2/91 */
!                 newsitm.itm_cid = 0;
!               cp1 = chop_str_plus(bufptr,'\n');
!                 sscanf(bufptr,"221 %d %s",&newsitm.itm_num,subject);
!                 util_idcpy(newsitm.itm_id,subject);
!                 while (cp1) {
!                 cp2 = chop_str_plus(cp1,'\n');
!                 if ((cp3 = chop_str_plus(cp1,':')) != 0) {
!                     while (isspace(*cp3)) ++cp3;
!                     lower_case(cp1);
!                     if (!strcmp(cp1,"subject")) util_subjcpy(newsitm.itm_title
,cp3);
!                     else if (!strcmp(cp1,"from")) util_fromcpy(newsitm.itm_fro
m,cp3);
!                     else if (!strcmp(cp1,"message-id")) util_idcpy(newsitm.itm
_id,cp3);
!                     else if (!strcmp(cp1,"lines")) {
!                       if (sscanf(cp3,"%d",&(newsitm.itm_lines)) == 1)
!                         newsitm.itm_flags |= NEWS_M_LINESVALID;
!                       }
!                     else if (!strcmp(cp1,"date"))
!                       newsitm.itm_postdate = newsitm.itm_recvdate = parse_usen
et_date(cp3);
!                   }
!                   cp1 = cp2;
!                   }
!                 if (gap->grp_iasize == gap->grp_count) {
!                   gap->grp_iasize += 10;
!                   gap->grp_ia = realloc(gap->grp_ia,(gap->grp_iasize + 1) * (s
izeof newsitm));
!                   }
!                 gap->grp_ia[++gap->grp_count] = newsitm;
                  }
                }
!           }
            }
          }
        if (gap->grp_unread > gap->grp_count) gap->grp_unread = gap->grp_count;
***************
*** 2696,2701 ****
--- 2817,2825 ----
  #if NNTP_USE_XHDR
    /* we should only get this far if we're an nntp client and we used the
       XHDR command in map_items() */
+ /*+ V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
+   err_line("DEBUG: Asking for Message-ID's via XHDR");
+ /*- V6.1B8 - 4-Sep-1993 - tmk - Temporary debug support for XOVER */
    if (strcmp(ga[g]->grp_name,nntp_currgroup)) {
      sprintf(err_oline,"GROUP %s",ga[g]->grp_name);
      if (nntp_one_call(nntp_node,nntp_proto,err_oline,&nntp_line) != 211)

        Terry Kennedy           Operations Manager, Academic Computing
        terry@spcvxa.bitnet     St. Peter's College, Jersey City, NJ USA
        terry@spcvxa.spc.edu    +1 201 915 9381
