INFO-VAX Sat, 22 Mar 2008 Volume 2008 : Issue 163 Contents: Re: Bracknell (Berkshire, UK) and VMS Re: Bracknell (Berkshire, UK) and VMS RE: Bracknell (Berkshire, UK) and VMS Re: Divining the full pathname of a file, all logicals translated Hitachi 500G SATA drive in a PWS 600au Re: Hitachi 500G SATA drive in a PWS 600au Re: Hobbyist Licenses Re: Hobbyist Licenses Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) RE: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) RE: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Re: Too many files in one directory (again) Tripwire on OpenVMS Best Practices? Re: VMS Mail translates incoming tilde character into a dollar sign. RE: What sysgen param needs to be changed? ---------------------------------------------------------------------- Date: Fri, 21 Mar 2008 11:32:25 -0700 (PDT) From: Didier_Toulouse Subject: Re: Bracknell (Berkshire, UK) and VMS Message-ID: <014739ce-2a38-42e9-9426-5087afca500b@a1g2000hsb.googlegroups.com> http://www.jobserve.com/W47B7E1939656F574.job ------------------------------ Date: 21 Mar 2008 18:16:24 -0600 From: kuhrt.nospammy@encompasserve.org (Marty Kuhrt) Subject: Re: Bracknell (Berkshire, UK) and VMS Message-ID: In article <551107f2-75b5-4e3b-9316-96e28408c22a@e23g2000prf.googlegroups.com>, IanMiller writes: > Big HP building at Bracknell (Amen Corner) just near the ski slope and > the ice rink. Ski slopes? In England? ------------------------------ Date: Fri, 21 Mar 2008 22:46:06 +0000 From: "Main, Kerry" Subject: RE: Bracknell (Berkshire, UK) and VMS Message-ID: > -----Original Message----- > From: Marty Kuhrt [mailto:kuhrt.nospammy@encompasserve.org] > Sent: March 21, 2008 8:16 PM > To: Info-VAX@Mvb.Saic.Com > Subject: Re: Bracknell (Berkshire, UK) and VMS > > In article <551107f2-75b5-4e3b-9316- > 96e28408c22a@e23g2000prf.googlegroups.com>, IanMiller > writes: > > Big HP building at Bracknell (Amen Corner) just near the ski slope > and > > the ice rink. > > Ski slopes? In England? In Canada, we call them snow banks .. :-) Regards Kerry Main Senior Consultant HP Services Canada Voice: 613-254-8911 Fax: 613-591-4477 kerryDOTmainAThpDOTcom (remove the DOT's and AT) OpenVMS - the secure, multi-site OS that just works. ------------------------------ Date: 21 Mar 2008 21:33:26 +0100 From: peter@langstoeger.at (Peter 'EPLAN' LANGSTOeGER) Subject: Re: Divining the full pathname of a file, all logicals translated Message-ID: <47e429a6$1@news.langstoeger.at> In article <692bd5d0-fcdc-4df7-9279-1955a49d7022@e23g2000prf.googlegroups.com>, Ken.Fairfield@gmail.com writes: > $ string = string - "][" - "]<" - ">[" - "><" > >That pretty much covers all cases. And "subtracting" a substring >that doesn't exist from another string is a no-op, so no worries. :-) But you should worry about a directory with mixed brackets as result $ DIR DSA0:[SYS0.SYSCOMMON.SYSMGR> %DCL-W-DIRECT, invalid directory syntax - check brackets and other delimiters \DSA0:[SYS0.SYSCOMMON.SYSMGR\ $ DIR DSA0:[SYS0.SYSCOMMON.] Directory DSA0:[SYS0.SYSCOMMON.] ... So, better keep it string = string - "][" -- Peter "EPLAN" LANGSTOEGER Network and OpenVMS system specialist E-mail peter@langstoeger.at A-1030 VIENNA AUSTRIA I'm not a pessimist, I'm a realist ------------------------------ Date: 21 Mar 2008 18:07:07 -0600 From: kuhrt.nospammy@encompasserve.org (Marty Kuhrt) Subject: Hitachi 500G SATA drive in a PWS 600au Message-ID: Hi all, My wife works for an online photo company and they use a lot of disk space. They are currently up to about eight petabytes of storage. The windfall for me is that they get rid of a bunch of working disks fairly often so they can upgrade. She just brought home a Hitachi 500G SATA drive for me to play around with. Drooling like Homer Simpson over a doughnut, I thought it would be neat if I could put this in my PWS 600au. So I ask the group; is there a PCI SATA HBA available for this machine? If not, is there a way, with an adapter, I could hang a few of these off one of my current SCSI HBAs (like a Qlogic ISP1020 SCSI-2)? Any suggestions, or working examples, would be appreciated. Thanks, Marty ------------------------------ Date: 21 Mar 2008 23:36:16 GMT From: healyzh@aracnet.com Subject: Re: Hitachi 500G SATA drive in a PWS 600au Message-ID: Marty Kuhrt wrote: > Drooling like Homer Simpson over a doughnut, I thought it would > be neat if I could put this in my PWS 600au. So I ask the group; > is there a PCI SATA HBA available for this machine? If not, is > there a way, with an adapter, I could hang a few of these off one > of my current SCSI HBAs (like a Qlogic ISP1020 SCSI-2)? > Any suggestions, or working examples, would be appreciated. The only thing I can think of would be a Acard SCSI-to-SATA bridge. I for one would love to know how it works. I wouldn't mind replacing the 2 external Andataco 3-Drive JBOD boxes with 10,000RPM drives with 1-2 internal drives. As I'd like to move my XP1000 to something *QUIETER* and less power hungry! Then I'd just need to get the BA350 off of my VAXstation 4000/vlc. Zane ------------------------------ Date: Fri, 21 Mar 2008 17:34:30 -0500 From: BC Berry Subject: Re: Hobbyist Licenses Message-ID: On Wed, 19 Mar 2008 10:04:37 -0500, DanS wrote: >In article , > wrote: > >> How do I renew my VAX VMS hobbiest licenses. I tried using >> htttp://www.openvmshobbyist.com/register.php a couple of weeks ago. >> The site says the request was accepted and the licenses will be >> E-mailed but they never arrived. I verified my spam filter was not >> blocking the site and tried again this weekend with the same results. > >That's very strange - when I've renewed my hobbyist licenses, the >emails came immediately. Perhaps try it with a different email >address. > >Dan Apparently Bellsouth (uhhh AT&T) is eating the E_Mails. After several failed attempts, I used my hotmail address and got the renewals within seconds. -- E-mail address is invalid due to spam overflow - Please reply in the newsgroup. -- ------------------------------ Date: Fri, 21 Mar 2008 17:37:19 -0500 From: BC Berry Subject: Re: Hobbyist Licenses Message-ID: <10e8u310i4ndc0ag38ipmfg68rrfg7j6t9@4ax.com> On Thu, 20 Mar 2008 05:46:10 GMT, John Santos wrote: >In article , >nobody@spamcop.com says... >> How do I renew my VAX VMS hobbiest licenses. I tried using >> htttp://www.openvmshobbyist.com/register.php a couple of weeks ago. >> The site says the request was accepted and the licenses will be >> E-mailed but they never arrived. I verified my spam filter was not >> blocking the site and tried again this weekend with the same results. >> > >When do they expire? ISTR that if you try to renew too soon (more >than a month or so before expiration), you'll get exactly that >symptom. Also, if you are trying to register a second system with >the same info (serial number, etc.) as one you've already registered, >it will think the current registration is still active and not issue >a new PAK. So make sure all your SN's are unique, if registering >more than one system. (A single set of Layered Products PAKs suffices >for all your boxes, though.) For an emulated system such as a "VAX" >running on SIMH on a PC, etc., use the SN of the host box, or just >make one up if unobtainable. They had already expired. That's why I was trying to get renewals. The application says to enter NONE for the serial number of SIMH or other emulator. That's what I did. I submitted with another E-mail address and got the renewals within a couple of seconds. Looks like my ISP's spam processor doesn't like Montager. -- E-mail address is invalid due to spam overflow - Please reply in the newsgroup. -- ------------------------------ Date: Fri, 21 Mar 2008 14:17:44 -0400 From: JF Mezei Subject: Re: Too many files in one directory (again) Message-ID: <47e3fbe9$0$28139$c3e8da3@news.astraweb.com> Out of curiosity, why would creating 190,000 files in a VMS directory be significantly slower than on Unix ? What does Unix do (or not do) that makes such a difference ? ------------------------------ Date: Fri, 21 Mar 2008 14:21:08 -0400 From: JF Mezei Subject: Re: Too many files in one directory (again) Message-ID: <47e3fcb5$0$28139$c3e8da3@news.astraweb.com> Prior to issuing the TAR command to create 190,000 files, did you just CREATE/DIRECTORY or did you CREATE/DIRECTORY/ALLOCATION=xxxxxxx ? Would this have made a huge difference in the file creation speed ? Would it make sense to SET FILE mydir.dir;1/global=5000 (or whatever) ? ------------------------------ Date: Fri, 21 Mar 2008 14:25:32 -0400 From: JF Mezei Subject: Re: Too many files in one directory (again) Message-ID: <47e3fdbe$0$28139$c3e8da3@news.astraweb.com> In a situation where one utility (TAR) is creating a zillion files, would it make a big difference to have the following: TAR unpacks in mydir1.dir You create mydir2 through mydir200 You have a separate process which runs say every minute, and does a RENAME [.mydir1]*.* [.mydirX] where X increases with every run and goes back to 2 after 200. This way, all directories would be of more manageable size, and TAR would never need to add files to a humoungous directory. ------------------------------ Date: Fri, 21 Mar 2008 14:34:48 -0400 From: "Richard B. Gilbert" Subject: Re: Too many files in one directory (again) Message-ID: <47E3FFC8.2050203@comcast.net> JF Mezei wrote: > Out of curiosity, why would creating 190,000 files in a VMS directory be > significantly slower than on Unix ? > > What does Unix do (or not do) that makes such a difference ? Unix does not store the directory entries in ascending alphanumeric order! The VMS developers, rightly or wrongly, assumed some degree of sanity in the users. I cannot imagine keeping 190,000 files in a directory. Performance, with a mere 70,000 files in a directory, is something I wish I could forget; I once had the dubious privilege of cleaning up such a mess! Without some help from DFU, I might still be deleting files, ten years later!! A sane VMS user, rather than create 190,000 files, might consider an indexed sequential file to hold the information. Unix, of course, since it lacks the very concept of "records" does not offer indexed sequential files or access records by key. The only operating systems that do, to my knowledge, are VMS and IBM O/S 360/370/whatever came after that. ------------------------------ Date: Fri, 21 Mar 2008 18:56:45 +0000 From: "Main, Kerry" Subject: RE: Too many files in one directory (again) Message-ID: > -----Original Message----- > From: JF Mezei [mailto:jfmezei.spamnot@vaxination.ca] > Sent: March 21, 2008 2:18 PM > To: Info-VAX@Mvb.Saic.Com > Subject: Re: Too many files in one directory (again) > > Out of curiosity, why would creating 190,000 files in a VMS directory > be > significantly slower than on Unix ? > > What does Unix do (or not do) that makes such a difference ? May not be specific to this issue, but a few things in general come to mind, especially with high write activities: - different design philosophy using indexed files vs individual files - default system and process parameters are often much different - default write philosophy - UNIX write through (speed) vs OpenVMS write back (data safety). You can set OpenVMS default for some scenarios to be write back with RMS_SEQFILE_WBH (sysgen> help sys_p RMS_SEQFILE_WBH) - disk high-water marking is security default on OpenVMS i.e. zero blocks before allocating to the process requesting additional space Regards Kerry Main Senior Consultant HP Services Canada Voice: 613-254-8911 Fax: 613-591-4477 kerryDOTmainAThpDOTcom (remove the DOT's and AT) OpenVMS - the secure, multi-site OS that just works. ------------------------------ Date: Fri, 21 Mar 2008 15:00:07 -0400 From: JF Mezei Subject: Re: Too many files in one directory (again) Message-ID: <47e405d9$0$28131$c3e8da3@news.astraweb.com> Richard B. Gilbert wrote: > Unix does not store the directory entries in ascending alphanumeric > order! The VMS developers, rightly or wrongly, assumed some degree of > sanity in the users. So when I do an "ls" in Unix, it does an in memory sort of the directory before listing it ? I guess it was a case of providing better performance for DIR vs CREATE while Unix provides better performance for CREATE vs DIR ------------------------------ Date: Fri, 21 Mar 2008 14:45:45 -0500 (CDT) From: sms@antinode.org (Steven M. Schweda) Subject: Re: Too many files in one directory (again) Message-ID: <08032114454569_2020CE0A@antinode.org> From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?= > > I just pass this along as a reminder that there's still some room > > for improvement in dealing with cases like this which,... > > And some of them are under your control, such as using a large > enough /ALLOCATION=n on the CRE/DIR command. I guess that it > would also help to INIT the device with a large enought > /HEADERS=n to begin with. Probably would. Drawbacks of this (and several other potentially helpful suggestions) is that _I_ didn't create the directory, VMSTAR did. The archive entries look like "zip/020989f4d6c2f32768d0535c1815344d.zip". Now I could, in principle, analyze the whole "tar" archive, puzzle out the optimal directory characteristics, and pre-create the things myself rather than trusting VMSTAR to know how to create directories. How many people believe that a user should need to go through that ordeal to avoid this problem? > Even if 190.000 files "works" on VMS, it might not be what > VMS was primarily designed for... So, it couldn't be improved to handle this case better? > How large/small is each individual file ? Pretty small, I believe. > Would it be possible (with 4 Gb available) to > create a DECram disk and run the un-tar against > that ? May be, but it'd probably be tight. And this is all on a system which I don't keep powered on constantly. From: "Main, Kerry" > Just a WAG, but since you are doing mostly write activities, can we assume > That you have removed disk highwater marking and set OpenVMS to the file > system default That UNIX uses (write back) vs OpenVMS's file system default > (write through)? I didn't. I didn't realize how doomed I was until I stepped well into this tar-pit. Volume Status: ODS-5, subject to mount verification, protected subsystems enabled, file high-water marking, write-through caching enabled, hard links enabled. When I get a chance, I'll try to look into VMSTAR to see if it is (or could be) using SQO to help avoid highwater marking trouble. In the past, it only bothered me when allocating space for a big file, and I believe that the files themselves are all pretty small. Should I still worry? From: Hein RMS van den Heuvel > > [...] Even growing the allocation by, say, 25% > > instead of one block might pay off without a big penalty > > It will be growing by at least a disk cluster at a time. > I don't think it'll honor the SET RMS/EXT It's probably the default for 143374741 blocks (SEAGATE ST373405LW) ODS5: Cluster size 16 but directory size growth seems to be better than that: IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:23:00 IT$DKA100:[cert]zip.DIR;1 26272/26272 IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:23:04 IT$DKA100:[cert]zip.DIR;1 26273/26384 Noticable (ten second?) pause waiting for that "26273/26384". > MONI FILE would be interesting FILE SYSTEM CACHING STATISTICS on node IT 21-MAR-2008 15:18:54.22 CUR AVE MIN MAX Dir FCB (Hit %) 100.00 100.00 100.00 100.00 (Attempt Rate) 10.66 10.22 10.00 10.66 Dir Data (Hit %) 70.00 70.66 70.00 72.00 (Attempt Rate) 6701.00 6738.11 6330.33 7183.00 File Hdr (Hit %) 89.00 88.66 88.00 89.00 (Attempt Rate) 9.66 9.33 9.00 9.66 File ID (Hit %) 100.00 100.00 100.00 100.00 (Attempt Rate) 1.00 1.00 1.00 1.00 Extent (Hit %) 100.00 100.00 100.00 100.00 (Attempt Rate) 1.00 1.00 1.00 1.00 Quota (Hit %) 0.00 0.00 0.00 0.00 (Attempt Rate) 0.00 0.00 0.00 0.00 Bitmap (Hit %) 0.00 0.00 0.00 0.00 (Attempt Rate) 0.00 0.00 0.00 0.00 IT $ sho mem /cac System Memory Resources on 21-MAR-2008 15:21:07.91 Extended File Cache (Time of last reset: 20-MAR-2008 05:59:48.57) Allocated (GBytes) 1.48 Maximum size (GBytes) 1.99 Free (GBytes) 0.00 Minimum size (GBytes) 0.00 In use (GBytes) 1.48 Percentage Read I/Os 22% Read hit rate 78% Write hit rate 0% Read I/O count 75035 Write I/O count 258987 Read hit count 58781 Write hit count 0 Reads bypassing cache 81 Writes bypassing cache 1521 Files cached open 548 Files cached closed 18757 Vols in Full XFC mode 0 Vols in VIOC Compatible mode 4 Vols in No Caching mode 0 Vols in Perm. No Caching mode 0 (Must be that lack of "Full XFC mode". Tee-hee-hee.) If I could find some dirt-cheap 2GB memory modules, I'd double it again to 8GB, but 4GB is all the budget would allow. > btw, do a semi random: > $ pipe dump/dir/blo=3D(star=3D10000,count=3D10) bad.dir | searc sys$pipe > "End of records" IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:33:24 IT$DKA100:[cert]zip.DIR;1 26376/26384 IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir | sear sys$ pipe "End of records" 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) IT $ pipe t ; dire /size = all /nohead /notrai IT$DKA100:[cert]zip.dir 21-MAR-2008 15:48:19 IT$DKA100:[cert]zip.DIR;1 26523/26608 IT $ pipe dump/dir/blo=(star=10000,count=10) IT$DKA100:[cert]zip.dir | sear sys$ pipe "End of records" 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) 012C End of records (-1) Seems pretty stable at "012C". From: koehler@eisner.nospam.encompasserve.org (Bob Koehler) > Can you put a listing in a file using tar -t and then break down > the output into multiple tar passes into multiple directories? And how should I specify the division of labor to VMSTAR? > I sure think I'd try that if I were in your situation. A little > time editing to create a script might save a great many hours. Complaining can be more fun than working, and (in my dreams) could be more likely to lead to some actual improvement than quietly finding some annoying work-around. From: JF Mezei > TAR unpacks in mydir1.dir > You create mydir2 through mydir200 > > You have a separate process which runs say every minute, and does a > RENAME [.mydir1]*.* [.mydirX] where X increases with every run and goes > back to 2 after 200. Or I could repackage the stuff in smaller pieces on the HP-UX system, or I could try breaking NFS by getting the files served from the HP-UX system, ... Many things are possible. Most of them, I claim, shouldn't be needed. From: "Richard B. Gilbert" > A sane VMS user, rather than create 190,000 files, might consider an > indexed sequential file to hold the information. [...] I didn't design this exercise. Someone else created a dump-truck-full of Zip (and other compress/archive) test files, mostly diddled in creative ways (they say) in an attempt to find problems in various expand/extract software. The packaging was probably done on a Linux system, where, I'd assume, no one saw any particular problems with the scheme. I just thought that it might be productive to run the stuff through the next version of UnZip to see if we had any problems on VMS. ------------------------------------------------------------------------ Steven M. Schweda sms@antinode-org 382 South Warwick Street (+1) 651-699-9818 Saint Paul MN 55105-2547 ------------------------------ Date: Fri, 21 Mar 2008 22:02:35 +0000 From: "Main, Kerry" Subject: RE: Too many files in one directory (again) Message-ID: > -----Original Message----- > From: Steven M. Schweda [mailto:sms@antinode.org] > Sent: March 21, 2008 3:46 PM > To: Info-VAX@Mvb.Saic.Com > Subject: Re: Too many files in one directory (again) > [snip ..] > > From: "Main, Kerry" > > > Just a WAG, but since you are doing mostly write activities, can we > assume > > That you have removed disk highwater marking and set OpenVMS to the > file > > system default That UNIX uses (write back) vs OpenVMS's file system > default > > (write through)? > > I didn't. I didn't realize how doomed I was until I stepped well > into this tar-pit. > > Volume Status: ODS-5, subject to mount verification, protected > subsystems > enabled, file high-water marking, write-through caching enabled, > hard > links enabled. > > When I get a chance, I'll try to look into VMSTAR to see if it is (or > could be) using SQO to help avoid highwater marking trouble. In the > past, it only bothered me when allocating space for a big file, and I > believe that the files themselves are all pretty small. Should I still > worry? > > Might not be applicable, but perhaps worth a try to see if the process appears to speed up some: Disk high water marking can be changed dynamically without dismounting the drive, so while you might be afraid to stop the job while it is running, you should be able to change it without impacting the job by entering: $ set volume $1$duax /nohighwater The same is true for modifying the sysgen parameter RMS_SEQFILE_WBH Reference: sysgen> help sys_p RMS_SEQFILE_WBH (Alpha and I64) RMS_SEQFILE_WBH can enable the RMS write behind feature as a system default for any unshared sequential disk file if the file is opened for image I/O with write access specified. [snip ..] Regards Kerry Main Senior Consultant HP Services Canada Voice: 613-254-8911 Fax: 613-591-4477 kerryDOTmainAThpDOTcom (remove the DOT's and AT) OpenVMS - the secure, multi-site OS that just works. ------------------------------ Date: Fri, 21 Mar 2008 19:50:44 -0500 From: "P. Thompson" Subject: Re: Too many files in one directory (again) Message-ID: On Fri, 21 Mar 2008, Richard B. Gilbert wrote: > A sane VMS user, rather than create 190,000 files, might consider an > indexed sequential file to hold the information. Unix, of course, since > it lacks the very concept of "records" does not offer indexed sequential > files or access records by key. The only operating systems that do, to > my knowledge, are VMS and IBM O/S 360/370/whatever came after that. I believe the OS HP most recently deprecated, MPE/IX, also featured similar record formats. I encountered this when editing a system file with MPE's, port of "vi" (which I was more comfortable with than the EDIT/EDT style editor MPE uses by default) and ended up corrupting the file requiring the MPE equivant of CONVERT/FDL to put it back. ------------------------------ Date: 21 Mar 2008 22:49:55 -0600 From: Kilgallen@SpamCop.net (Larry Kilgallen) Subject: Re: Too many files in one directory (again) Message-ID: In article <08032114454569_2020CE0A@antinode.org>, sms@antinode.org (Steven M. Schweda) writes: > From: =?ISO-8859-1?Q?Jan-Erik_S=F6derholm?= > >> > I just pass this along as a reminder that there's still some room >> > for improvement in dealing with cases like this which,... >> >> And some of them are under your control, such as using a large >> enough /ALLOCATION=n on the CRE/DIR command. I guess that it >> would also help to INIT the device with a large enought >> /HEADERS=n to begin with. > > Probably would. Drawbacks of this (and several other potentially > helpful suggestions) is that _I_ didn't create the directory, VMSTAR > did. The archive entries look like > "zip/020989f4d6c2f32768d0535c1815344d.zip". Now I could, in principle, > analyze the whole "tar" archive, puzzle out the optimal directory > characteristics, and pre-create the things myself rather than trusting > VMSTAR to know how to create directories. The nice part of that approach is that from the general description of the problem it would seem you would not have to create many directories :-) ------------------------------ Date: Fri, 21 Mar 2008 21:34:32 -0500 (CDT) From: sms@antinode.org (Steven M. Schweda) Subject: Re: Too many files in one directory (again) Message-ID: <08032121343250_2020CE0A@antinode.org> > From: "Main, Kerry" > > > Just a WAG, but since you are doing mostly write activities, can we assume > > That you have removed disk highwater marking and set OpenVMS to the file > > system default That UNIX uses (write back) vs OpenVMS's file system default > > (write through)? > > I didn't. I didn't realize how doomed I was until I stepped well > into this tar-pit. > > Volume Status: ODS-5, subject to mount verification, protected subsystems > enabled, file high-water marking, write-through caching enabled, hard > links enabled. > > When I get a chance, I'll try to look into VMSTAR to see if it is (or > could be) using SQO to help avoid highwater marking trouble. In the > past, it only bothered me when allocating space for a big file, and I > believe that the files themselves are all pretty small. Should I still > worry? For the record, even before I started fiddling with it (V3.4-1), VMSTAR was pre-allocating disk space for extracted files ("fil_fab.fab$l_alq = (unsigned long) ((bytes+511)/512);"), and the edition which I've been using (V3.5-pre2, "http://antinode.org/ftp/vmstar/v3r5_pre2_src.zip") also sets the "sequential access only" flag ("fil_fab.fab$l_fop |= FAB$M_SQO;"), so, pending a good counter-argument, I'll believe that highwater marking shouldn't be bothering the application code. (If it's hampering the OS and/or file system, then I figure that I can pass along the blame for that to someone else.) Lacking an override from the user, it should also (now) be using 127 for the multi-block count ("fil_rab.rab$b_mbc"), and using two buffers ("fil_rab.rab$b_mbf"), and setting write-behind ("fil_rab.rab$l_rop |= RAB$M_WBH;") on the extracted files. (This code now looks remarkably like the corresponding code in UnZip. A coincidence, no doubt.) I'll admit that extracting an archive using VMSTAR has always seemed much slower than using any real "tar" on a UNIX system, which is one reason I started adding this code to it in the first place. I'll also admit that my changes probably didn't make a world of difference in that situation. (I will claim that it handles symlinks and GNU "tar" long names better than it did, though.) ------------------------------------------------------------------------ Steven M. Schweda sms@antinode-org 382 South Warwick Street (+1) 651-699-9818 Saint Paul MN 55105-2547 ------------------------------ Date: Fri, 21 Mar 2008 21:57:50 -0700 (PDT) From: Hein RMS van den Heuvel Subject: Re: Too many files in one directory (again) Message-ID: <17822d27-7fff-45b1-b054-5ba39e150aa2@8g2000hse.googlegroups.com> On Mar 21, 3:45=A0pm, s...@antinode.org (Steven M. Schweda) wrote: > From: =3D?ISO-8859-1?Q?Jan-Erik_S=3DF6derholm?=3D > > > > =A0 =A0I just pass this along as a reminder that there's still some ro= om > > > for improvement in dealing with cases like this which,... > > Even if 190.000 files "works" on VMS, it might not be what > > VMS was primarily designed for... > > =A0 =A0So, it couldn't be improved to handle this case better? Yes. Maybe, Just maybe, Andy will get to release a b-tree based implementation. > > create a DECram disk and run the un-tar against that ? > > =A0 =A0May be, but it'd probably be tight. =A0And this is all on a system > which I don't keep powered on constantly Create an LD container first. Copy the container to a the real disk. > =A0 =A0I didn't. =A0I didn't realize how doomed I was until I stepped well= into this tar-pit. Oh yeah. Classic. Can't kill it now.. it must almost be done. :-) Just wait till you get to delete them! > > > [...] =A0Even growing the allocation by, say, 25% > > > instead of one block might pay off without a big penalty Mark Hopkins whisperred to me "1) A directory extends by the maximum of 100 blocks or 1/2 the current file size (I just looked in the code). I'm pretty sure that this is rounded up to the next cluster factor. None of the default extends are honored here. 2) If someone hasn't already suggested it, increase acp_maxread from 32 to 64. 3) Turn on RMS_SEQFILE_WBH if possible." > Noticable (ten second?) pause waiting for that "26273/26384". That suggests that there was no free space found next to the current allocation and the directory was copied ACP_MAXREAD blocks at a time I made the DUMP/HEAD | grep LBN suggestion to check just that. Pre-allocation would have prevented that delay. Suspend the job (if not done by now), use DFU to expand, resume. > > MONI FILE would be interesting > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0FILE SYSTEM CACHING STA= TISTICS > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0CUR =A0 =A0 =A0 =A0AVE =A0 =A0 =A0 =A0MIN =A0 =A0 =A0 =A0MAX > =A0 =A0 Dir Data =A0 (Hit %) =A0 =A0 =A0 =A0 =A0 =A0 =A0 70.00 =A0 =A0 =A0= 70.66 =A0 =A0 =A070.00 =A0 =A0 =A072.00 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0(Attempt Rate) =A0 =A0 =A06701.00 =A0 =A067= 38.11 =A0 =A06330.33 =A0 =A07183.00 Yikes! Is is doing some readdir before doing the create files? > IT $ pipe dump/dir/blo=3D(star=3D10000,count=3D10) IT$DKA100:[cert]zip.dir= | sear sys$ > pipe "End of records" > 012C =A0End of records (-1) : > 012C =A0End of records (-1) > Seems pretty stable at "012C". So all blocks are nicely filled, suggesting files are nicely added in sequence. In that case actuall adding of files 100..200 should be no slower than 100100 .. 100200. But the directory growth / reallocation will be getting progressively worse as you noticed. > =A0 =A0Complaining can be more fun than working, and (in my dreams) could = be > more likely to lead to some actual improvement than quietly finding some > annoying work-around. Agreed! Too many poor souls out there do not even have the first clue to work around this. > From: JF Mezei > > > TAR unpacks in mydir1.dir > > You create mydir2 through mydir200 > > > You have a separate process which runs say every minute, and does a > > RENAME [.mydir1]*.* [.mydirX] =A0where X increases with every run and go= es > > back to 2 after 200. clueless > =A0 =A0Many things are possible. =A0Most of them, I claim, shouldn't be ne= eded. Agreed. > From: "Richard B. Gilbert" > > > A sane VMS user, rather than create 190,000 files, might consider an > > indexed sequential file to hold the information. =A0[...] Or an indexed file to hold name/fid tupples for directory less files. :-) JF>> Out of curiosity, why would creating 190,000 files in a VMS directory be significantly slower than on Unix ? Because it uses a writeback model, not write through. That works most of the time. If 10 entries are made to a directory block, VMS will do 10 actual write IO. Your typical Unix will just do 1 IO, or none. Depends on the update deamon. Unix'es also have a variety of filesystems, many log based, making it a little safer. JF> did you CREATE/DIRECTORY/ALLOCATION=3Dxxxxxxx ? JF> Would this have made a huge difference in the file creation speed ? Yes, with the evidence currently in play. JF> Would it make sense to SET FILE mydir.dir;1/global=3D5000 (or whatever) ? Absolutely not. For starters, directories do use the normal RMS buffer caches. But if they did... Global Buffers cache buffers at certain VBN location. Well, after a few not-at-the-end inserts all blocks are shuffled up. So the contents of VBN 10 moves to VBN 11 and so. Not good for a VBN tag cache. JF> So when I do an "ls" in Unix, it does an in memory sort of the directory before listing it ? Unix has these 'inodes' which are like a mini file header, but they live in the directory. They hold the 'stat' info: byte-size, dates, owner,... So for unix is it trivial to list sizes or present sorted by date. OpenVMS has to do 1 IO per file for attributes other than name or file- id. This IOs may be cached, but still. Steven>> it should also (now) be using 127 for the multi-block count ("fil_rab.rab$b_mbc"), and using two buffers ("fil_rab.rab$b_mbf"), and setting write-behind ("fil_rab.rab$l_rop | =3D RAB$M_WBH;") on Some argue to make it a multiple of 4 (124) to make live easier on certain storage types (EVA Raid-5). Others recommend a multiple of 16 (112) to make life easier for the XFC (16 block cache lines) RMS WHB is a simple to use way to get concurrent IO going and more often than not well worth the CPU price. If CPU usage becomes a concern, then self-managed $WRITE, with $RAB64 may become desirable. For even less CPU time, go down to QIO and for the least possible CPU overhead, $IO_PERFORM. 2 buffers should be just fine. 4 often helps a little more. Beyond that I find rapidly diminishing returns. Cheers, Hein. ------------------------------ Date: Fri, 21 Mar 2008 13:52:41 -0700 (PDT) From: rfc2307@gmail.com Subject: Tripwire on OpenVMS Best Practices? Message-ID: <75da0787-9aa4-4eef-96eb-8d9ae220f1cc@s8g2000prg.googlegroups.com> Good Morning, I'm looking for any information on best practices on what to monitor, that is Operating System specific on the OpenVMS platform. Any information or lessons learned would be appreciated. Please reply to rfc2307@gmail.com Thanx in advance for all your assistance ------------------------------ Date: 21 Mar 2008 17:51:20 -0600 From: kuhrt.nospammy@encompasserve.org (Marty Kuhrt) Subject: Re: VMS Mail translates incoming tilde character into a dollar sign. Message-ID: In article <64hl7oF2c3h0sU1@mid.individual.net>, billg999@cs.uofs.edu (Bill Gunshannon) writes: > In article <47e32132$1$23906$c3e8da3@news.astraweb.com>, > JF Mezei writes: >> Phillip Helbig---remove CLOTHES to reply wrote: >> >>> This really shows how quality control has gone down the drain. >> >> The only thing of value VMS still has is the clustering software. The >> rest is all "legacy". When HP retires VMS, if the clustering is still >> state of the art, they may get a few pennies giving it to Microsoft (again). >> >> The rest doesn't need to have much quality control or development. > > Considering the internal differences between VMS and everything > else, not even the clustering code would be worth anything to anybody > else. > M$lop had the clustering code and couldn't make it work. I saw some demos of NT clustering, but outside of tradeshow razzmatazz I haven't seen it work. (And for all I know, and knowing how M$lop works, the tradeshow demo was probably just a well timed PowerPoint with no connection to a real system doing failover). ------------------------------ Date: Fri, 21 Mar 2008 13:48:11 -0400 From: "Hank Vander Waal" Subject: RE: What sysgen param needs to be changed? Message-ID: <011101c88b7b$bbde85a0$6500a8c0@dellxp30> Remote file opens do not count toward your vms licenses do they ?? I have found that by increasing the ncp alias file limit I could get more people accessing the system. -----Original Message----- From: Bob Koehler [mailto:koehler@eisner.nospam.encompasserve.org] Sent: Friday, March 21, 2008 3:04 PM To: Info-VAX@Mvb.Saic.Com Subject: RE: What sysgen param needs to be changed? In article <010201c88b66$2c3c8d30$6500a8c0@dellxp30>, "Hank Vander Waal" writes: > Bob & all, > I have several users on the remote site that are working its when I get more > than a couple of them, and each one opens about 15 files, that I get the > error. I am not sure if it is a user setting or a SYSGEN setting? > The netserver.log file does not report any errors > Do you by any chance have a limited number of users in the VMS license for the remote system? If not, I'd check the pool and page file, using SHOW MEMORY. ------------------------------ End of INFO-VAX 2008.163 ************************