This discussion is being used to collect information on the history and evolution of the MTS file system. It will be good to capture more of this information before we all forget. |
Discussions >
Evolution of the MTS file system
Internal documentation for the MTS file system
There is a 100+ page document that gives a quite complete description of the format and data structures used by the MTS file system at UM as of D6.0 in February 1988. See: http://deepblue.lib.umich.edu/bitstream/handle/2027.42/79607/MTSDiskInformation-Apr1987.pdf
The documentation includes:
|
Some IBM compatiable disk drive history, not specific to MTS
IBM Storage photo album (punched cards, disks, magnetic tapes, data cell, fixed head file, drum) Roger Broughton's (NCL) article(s) on DASD |
File system development timeline
|
Disk hardware used at U-M
On Jun 8, 2014, at 1:08 AM, Mike Alexander wrote: I do know the answers to some of these questions (as I'm sure Gary does too). Comments appear below. They were somewhat hastily entered so may have mistakes. Mike --On June 7, 2014 at 9:28:49 PM -0400 Jeff Ogden <jco@umich.edu> wrote: And what was the hardware evolution of the disks? Did MTS ever use 2311s? I'm sure that 2314s were used. I assume that that was followed by 3330s, 3330-IIs, and then 3350s. Were 3350s the first non-removable disks (ignoring the 2301 drum and 2305 fixed head file) we used? We had various brand-X disks (ITEL, Memorex, Amdahl, ...), but I think they were all compatible with some model of IBM disk. Were the Amdahl disks the first to have a native mode of their own or at least the first to have a native mode that we used? The first disks were 2311s. I think we used all the others you mention as well as a few oddball ones. Do you remember the super disk with for spindles and one actuator in a large case. MTS drove it so hard that it essentially melted and we were down for several days while the vendor (I forget now who it was) flew in a new one. These disks all emulated IBM disks, but some also had a native mode which MTS could use. This wasn't common, and I can't remember the details, but I'm sure there are tables in the distributions that would fill in some of these blanks. Did I miss anything? What came after the Amdahl disks? More IBM disks, probably 3390s of some flavor. By then we were very definitely an all-IBM shop. On Jun 8, 2014, at 1:23 PM, Mike Alexander wrote in response to several comments and questions from Jeff: On Jun 8, 2014, at 10:34 AM, Jeff Ogden wrote: Thanks Mike. . . . I know the file routines supported FBA [fixed block architecture] devices. Did U-M ever have any FBA disks? Yes, we had support for several FBA devices. It wasn’t hard since the file system used a fixed block size by then. Now that you mention it, I do remember the "super" disk. I think it was a 8800 from STC. See: History: Milestones in HDD Capacity A 1977 Computer World article on the STC Superdisk: http://books.google.com/books?id=gLYekwssyhsC&pg=PA44&lpg=PA44&dq=stc+superdisk&source=bl&ots=g55LG6CirG&sig=FaLIN0UWUB_jGKtTESBL41cA-MA&hl=en&sa=X&ei=I1eUU_TbLpSzyATtgIL4DA&ved=0CDUQ6AEwAg#v=onepage&q=stc%20superdisk&f=false An October 1973 ad for the Super Disk: http://books.google.com/books?id=qYmeRiv3ZxgC&pg=PT4&lpg=PT4&dq=stc+superdisk&source=bl&ots=TaXWwBRsPy&sig=2Nhnk7a3EKaHy4l5kKpFAgmovmo&hl=en&sa=X&ei=I1eUU_TbLpSzyATtgIL4DA&ved=0CDcQ6AEwAw#v=onepage&q&f=false That same issue of Computerworld has an article about the Superdisk on page 1. There were two models, the largest of which had a capacity of 800 MB, which was quite large for the time. Mike |
Disk formats
Original comments and questions from Jeff are in black. Replies from Mike in blue. On Jun 8, 2014, at 1:08 AM, Mike Alexander wrote: I do know the answers to some of these questions (as I'm sure Gary does too). Comments appear below. They were somewhat hastily entered so may have mistakes. Mike --On June 7, 2014 at 9:28:49 PM -0400 Jeff Ogden wrote: Another item of historical interest is the evolution of the format of the MTS file system on disk (or data cell). I came across this as part of the article "New Disk System to be Installed" in the 9 May 1973 issue of the CC Newsletter: Although any physical record format for the new disks would be incompatible with that of the old ones, we have chosen to "page-format" the new disks in order to eliminate, hopefully, even this minor incompatibility (the need to regenerate saved note/point information of sequential files) when acquiring new hardware in the future. So, it looks like MTS disks at U-M became page formatted over the weekend of June 30th. This of course begs the question of what they were before they were "page-formatted". It seems to be "track" oriented, but I'm unclear on the relationship between a page and a track. Was there a time when the block size for files on disk wasn't a page (4096 bytes)? The block size was never 4096 bytes until we went to page formatted disks. Before that the basic disk format was derived from OS/360, not TSS. Each track was formatted into a fixed number of records that filled the track up. On early disks I think it was one record per track, but later on there were more. Tracks were formatted the first time they were allocated (or maybe each they were allocated). before that they only contained a record 0 and no other records. And on a related, but slightly different topic: I know that the basic data structures on the MTS Volumes were based on formats from TSS. I know too that at some point we moved from something called VAM2 to VAMX. But in general I'm pretty vague on the details behind those names. I suspect that there is more information in some of the distribution notes. I suspect that some of the MTS Newsletters have more information as well. Was VAM2 the name for the original TSS format? Was there a VAM or VAM1? VAMX was our own elaboration of the VAM2 format, wasn't it? I think we continued to use the TSS DASDI and DUMP/RESTORE stand-alone utilities until the move to VAMX. There may have been a VAM or VAM1, but it never got outside IBM. VAM2 was the format used by TSS and adopted by MTS when it went to page sized records on the disk. At that time we also preformatted the disks so MTS could assume the page sized records were already there. VAMX was essentially the same as VAM2 except for one small, but important change. Page addresses on the disk were 32 bits. In VAM2 this consisted of 4 bits unused (except in DSCB pages), 12 bits volume number, and 16 bits page number. Disks got big enough that 16 bits wasn't enough for a page number so in VAMX the division was 4, 8, and 20. Other than that I think VAM2 and VAMX were the same. Did the original TSS formats include the catalog or just the volume label, PAT and relocation entries, and DSCBs? The catalog was entirely MTS's (mostly Gary's) invention. The label format is from OS/360 (mostly) while the DSCBs, PAT, and relocation entries are taken from TSS. On Jun 8, 2014, at 1:23 PM, Mike Alexander wrote in response to several comments and questions from Jeff: On Jun 8, 2014, at 10:34 AM, Jeff Ogden wrote: Thanks Mike. So, before page formatted disks, MTS file buffers were larger or smaller than 4096. Weird. I have 4096 baked into my brain. And before page formatted disks, file space was allocated in terms of tracks rather than pages or records? Yes, that’s why old FS tapes show file sizes in tracks, not pages. I know the file routines supported FBA [fixed block architecture] devices. Did U-M ever have any FBA disks? Yes, we had support for several FBA devices. It wasn’t hard since the file system used a fixed block size by then. Now that you mention it, I do remember the "super" disk. I think it was a 8800 from STC. See: History: Milestones in HDD Capacity A 1977 Computer World article on the STC Superdisk: http://books.google.com/books?id=gLYekwssyhsC&pg=PA44&lpg=PA44&dq=stc+superdisk&source=bl&ots=g55LG6CirG&sig=FaLIN0UWUB_jGKtTESBL41cA-MA&hl=en&sa=X&ei=I1eUU_TbLpSzyATtgIL4DA&ved=0CDUQ6AEwAg#v=onepage&q=stc%20superdisk&f=false An October 1973 ad for the Super Disk: http://books.google.com/books?id=qYmeRiv3ZxgC&pg=PT4&lpg=PT4&dq=stc+superdisk&source=bl&ots=TaXWwBRsPy&sig=2Nhnk7a3EKaHy4l5kKpFAgmovmo&hl=en&sa=X&ei=I1eUU_TbLpSzyATtgIL4DA&ved=0CDcQ6AEwAw#v=onepage&q&f=false That same issue of Computerworld has an article about the Superdisk on page 1. There were two models, the largest of which had a capacity of 800 MB, which was quite large for the time. Mike |
File types: Line files, Sequential files, Sequential with Line Number files
Line files were the first and only file type in the earliest days of MTS--On June 8, 2014 at 1:29:29 PM -0400 Jeff Ogden <jco@umich.edu> wrote: On Jun 8, 2014, at 4:03 PM, Mike Alexander replied: No. Line files were there from the very beginning, although with more restrictions on the size of a file and the length of a line as you've already pointed out. Line files existed even before paging. Sequential and sequential with line number files added to support the Data CellOn Jun 8, 2014, at 11:39 AM, Gary Pirkola <gpirkola@umich.edu> wrote: On Jun 8, 2014, at 1:23 PM, Mike Alexander wrote: Sequential files were invented for the data cell. That was the only reason we invented a separate file type. We figured that be eliminating random access (or at lest making it harder) we could avoid picking strips so often. Sequential files read and wrote pages sequentially which hopefully would keep the same strip active for longer. I’m not sure if there was anything special about sequential files on the data cell as opposed to on disks. --On June 8, 2014 at 1:29:29 PM -0400 Jeff Ogden <jco@umich.edu> wrote: On Jun 8, 2014, at 4:03 PM, Mike Alexander wrote: Yes. They were meant mainly to be a way to save line files on the data cell. We didn't allow you to create a line file there so if you wanted to save a line file on the cheaper data cell storage and preserve line numbers you needed to use a SEQWL file. Line files with lines longer than 255 bytes and larger than 255 pagesFrom page 2 of the U-M Computing Center Newsletter, Volume 5, Number 14, 24 September 1975:  |
Mistakes in the design of the MTS file system
This is part of a note from Mike Alexander: On Apr 29, 2014, at 1:12 PM, Mike Alexander wrote: We also made at least two mistakes in the design of the file system: - zero length writes delete lines so there can't be a zero length line in a file - fine names are only 16 characters long when there was room on disk [in the DSCB] for 44 characters. Mike |
Update to the MTS architecture article in Wikipedia (Draft)
On 7 June 2014 new sub-sections on "Shared files" and "File locking" were added to the "MTS Systems Architecture" article in Wikipedia. The new sub-sections are an update of the draft that follows with changes suggested by Gary Pirkola and Mike Alexander. -JeffOn May 25, 2014, at 12:03 AM, Jeff Ogden wrote: The "MTS System Architecture" article in Wikipedia doesn't currently (May 2014) talk about shared files or file locking and so I'm gathering information for an addition to that article. Here is a draft of a possible addition. Comments, suggestions, and corrections welcome. -Jeff Shared files Over time the sharing of files between MTS users evolved in four stages. Stage one allowed for limited file sharing, where public or library files (files whose names start with an asterisk) were readable by all users and all other files (user files) could only be accessed by their owners. Public files were owned and maintained by Computing Center staff members, so at this stage only Computing Center files were shared. Stage two allowed for limited file sharing, where the program *PERMIT could be used to (i) make a file read-only (RO) to the file’s owner and all other MTS users, (ii) make a file available for copying by members of the same project as the file’s owner using the program *COPY, or (iii) make a file available for copying by all other users using the program *COPY. As for stage one, by default owners had unlimited access to their own files and the files were not accessible to other users. Stage three allowed for “really shared files”, where the $PERMIT command or the PERMIT subroutine can be used to share a file in a variety of ways with lists of other users, projects, all other users, or a combination of these. The types of access that can be allowed are read, write-extend, write-change or empty, renumber or truncate, destroy, and permit. As for stages one and two, by default a user file is permitted with unlimited access for its owner and no access for others. A file’s owner’s access can also be changed, although an owner always retains permit access. The $FILESTATUS command or FILEINFO and GFINFO subroutines can be used to obtain a file’s permit status. Stage four added program keys (PKeys) to the list of things to which a file can be permitted. Thus files can be permitted to users, projects, all other users, program keys, or a combination of these. Program keys were associated with MTS commands and files, which allowed files to be permitted to specific programs or to specific MTS commands. Among other things this allowed the creation of execute-only or run-only programs in MTS. The PKEY subroutine can be used to shorten the program key of the currently running program or switch the program key of the currently running program to *EXEC and later restore the program key, allowing a program to voluntarily limit the access it has to files by virtue of its program key. File locking As part of “really shared files” (stage three), file locking was introduced to control simultaneous access to shared files between active MTS sessions (that is, between separate running tasks or processes). File locking does not limit or block access to files within a single MTS session (between command language subsystems or user programs running as part of the same MTS session). File locking in MTS is mandatory rather than advisory. Files are locked implicitly on first use of a particular type of access or explicitly using the $LOCK command or the LOCK subroutine. Files are unlocked implicitly when the last use of a file within a task is closed or explicitly using the $UNLOCK command or the UNLK subroutine. The $LOCKSTATUS command or LSFILE and LSTASK subroutines can be used to obtain a file’s or a task’s current lock status. A file’s lock status may be “not open and not locked”, “open”, “locked for read”, “locked for modify”, “locked for destroy”, “waiting for open”, “waiting for read”, “waiting for modify”, or “waiting for destroy”. Locking a file for modification also locks the file for reading and locking a file for destroying also locks the file for modification and reading. Any number of jobs can have a file locked for reading at any given time, but only one job can have a file locked for modification at any given time and then only if no job has the file locked for reading, or locked for destroying. Only one job can have a file locked for destroying at any given time, and then only if no job has the file open, locked for reading, or locked for modification. When an attempt to lock a file cannot be satisfied, the calling task will wait either indefinitely or for a specific period of time for another task to unlock the file, or until an attention interrupt is received. If the file cannot be locked, an error indicating this is returned. The file locking software detects deadlocks between tasks and returns an error indication. |
File sharing in early versions of MTS
E-mail to Gary asking about the stages of file sharing in MTS
From: Jeff Ogden Subject: file sharing in early versions of MTS Date: May 18, 2014 5:35:55 PM EDT To: Gary Pirkola Gary, am I remembering this early history correctly? The first of it was before my time at the CC. (Stage 1) In the beginning: Library (public) files are readable by all users and modifiable by people who know a special password (limited to CC staff).
User files are readable and modifiable by their owners and no one else. (Stage 2) Later: It became possible to "permit" user files to be readable (read-only or RO) by all other users. There was a program to do this, but I forget its name, was it *Permit? [yes, it was]
(Stage 3) And later still: Really shared files added file locking and the ability to permit files to ccids, project ids, and pkeys as well as initial substrings of ccids, project ids, and pkeys using the $Permit command in a wide variety of ways (read, write-change, write-expand, destroy, …).
File locking and deadlock detection was introduced.
I think that that is pretty much it. Were pkeys introduced at the same time as the ability to permit to ccids and project ids? Before file locking was introduced what, if anything, protected a file that was being read by some users and modified by its owner at the same time? I assume that needing a password to modify a library file was discontinued at stage (2) and from then on there was really nothing too different about library files and user files. I guess I could go read your IEEE paper, but I don't have a copy at home and am too lazy to go to the U-M Library. -Jeff Gary's replyFrom: Gary Pirkola Gary, am I remembering this early history correctly? The first of it was before my time at the CC.
I believe this is correct (Stage 2) Later:
My memory is not very clear about what was available/possible at this intermediate stage. Mike? Did we really have "read only" data files for example or just "run only" object files for users? (Stage 3) And later still:
Yes, this part is correct, I remember well :) I think that that is pretty much it. Were pkeys introduced at the same time as the ability to permit to ccids and project ids? Actually no. I just reread the paper John and I wrote called “Protection of Information in a General Purpose Time Sharing Environment” for a Computer Security and Integrity Conference in May, 1977, and it says that "the facility for sharing files among users and projects has been available for close to 4 years (since 1973?), whereas the facility for giving programs access to files has been available for only about one year (since 1976?)” Before file locking was introduced what, if anything, protected a file that was being read by some users and modified by its owner at the same time? Good question. Mike? Was there any way for the “lower level system” or higher level MTS command interface to detect and prevent this? There was nothing in the file system proper that I remember before really shared files, but I can’t image that we would have just let this happen? If we only had run only object files, this would not have been quite so serious a problem. I assume that needing a password to modify a library file was discontinued at stage (2) and from then on there was really nothing too different about library files and user files. I guess I could go read your IEEE paper, but I don't have a copy at home and am too lazy to go to the U-M Library.
Gary Some information from the U-M Computing Center NewsletterFrom: Jeff Ogden     From: Jeff Ogden  From the CC Newsletter, Vol. 2, No. 9, 30 June 1972, page 6:    Another reply from GaryOn May 21, 2014 10:15:05 AM -0400 Gary Pirkola wrote:Well that certainly answers my question about what you could do with the *permit program. Thanks Jeff. So now I’m even more intrigued to remember how we prevented an owner from modifying a data file while other users were reading it. Seems like a recipe for an inconsistent file down at the file buffer level otherwise. Gary A reply from Mike about *PERMIT and *COPY (stage 2)On May 24, 2014, at 8:06 PM, Mike Alexander wrote:It's part of the answer, but not quite all of it. I've attached two sections from the 3rd edition (November 1969) of Volume 2 for *PERMIT and *COPY. If you're really interested, I've also attached the source for both of them from D2. I also have extracted the source for OPEN, FCBEQU and CATEQU from D2 if you want them. Although the writeup for *Permit talks about both a "how" and a "who" parameter, it really only takes one parameter other than the file name. If present, it must be one of "RO", "ALL", "NONE", or "PRJNO" where "ALL" is the default. RO makes the file read only to everyone, including the owner. "NONE", of course, makes it unlimited to the owner and no access for anyone else. "ALL" and "PRJNO" interact with *COPY. If either of these is given then others (either all others or in the same project) can copy the file using *COPY but not access it otherwise. I think the owner still has write access but I didn't verify that. Hence the snippet from the Newsletter that maps *PERMIT to Really Shared Files is a bit misleading. "ALL" maps to U Owner, R All, but they aren't really the same since *Permit's "ALL" only allowed access by *COPY. *PERMIT[From MTS Volume 2, Public File Descriptions, Third Edition, November 1969.] Contents: The object module of a program to allow users to permit their files to be accessed by other users. Purpose: To allow files to be shared between user ID numbers. Usage: The program is invoked by a $RUN command. Logical I/O Units Referenced: GUSER - filename, access type and/or sharers (unless parameters are given). Description: Three parameters separated by blanks are required as input for each file being permitted. They are: What How Who What = the name of the file to be shared. How = the type of sharing access allowed. Currently Read-Only is the only type of access allowed and is designated by RO. This parameter should be omitted if Read-Only is not desired. Notes: 1. Designation of a file by a user as Read-Only shared allows other users to access that file directly (without having to copy it via the *COPY program). This file may then only be read - it may not be changed, even by the owner of the file. 2. If the owner of the file "X" has SIGNON ID ABCD, then the filename ABCD:X will refer to this file and may be used anywhere a filename may be used provided that the usage is a read-type usage and provided that user ABCD has permitted file "X" as Read-Only shared. 3. If one has a file whose name has a colon as one of the first five characters, he must refer to it using XXXX:filename where XXXX is his SIGNON ID ,e.g., if user ABCD has file 0:1 he must refer to it as ABCD:0:1. 4. The file should be completely prepared and checked before being permitted as RO. If changes must be made later, it must be unpermitted (permitted with NONE) before the changes can be made. This must not be done when someone else is using it. Pre- sently there is no check made to determine whether someone is accessing the file when it is being unpermitted. Who = the sharers who are allowed to access the file. Currently three classes of sharers are allowed. These are: ALL - meaning all users (default). PRJNO - meaning all users who have the same project number as the permitting user. The project number appears as "charge number" on the tail-sheet of batch runs. It is not (in most cases) the same as the SIGNON ID. Students in a class, or other groupings of users, are assigned dif- ferent SIGNON ID's but the same project number. NONE - meaning no one else. Notes: 1. Current restriction: If RO is speci- fied, ALL is assumed. 2. If PRJNO or ALL is specified, the program *COPY may be run to make a copy of the file under a new user ID. 3. When a file is created, it has access type NONE. Examples: $RUN *PERMIT PAR=MYFILE RO permits the file to be Read-Only shared by all users. $RUN *PERMIT PAR=MYFILE2 permits the file to be copied by all users via *COPY. To permit several files, the parameter field should be omitted. The program will request the information via logical I/O unit GUSER. In either terminal or batch mode, the program will read lines, each having a file name, an access type and/or a sharing class, until an end of file is read. *COPY[From MTS Volume 2, Public File Descriptions, Third Edition, November 1969.]Contents: The object module of a program which copies permitted files (line or sequential) from one user account to another. Usage: The program is invoked by a $RUN command specifying *COPY as the location of the object module. Prototype: $RUN *COPY [SPUNCH=FDname PAR=userid:FDname] Examples: To copy one file: $RUN *COPY SPUNCH=-T PAR=STAT:NEWS To copy a series of files (user input is in lower case): $run *copy EXECUTION BEGINS ENTER OUTPUT FDNAME -x ENTER "SIGNON:FILE-NAME" stat:news ENTER OUTPUT FDNAME -y ENTER "SIGNON:FILE-NAME" xxxx:hisfile INPUT FILE NOT FOUND $endfile EXECUTION TERMINATED Logical I/O Units Referenced: SPUNCH - Output of the files being copied. GUSER - Input of "to" and "from" file names if no PAR= field was specified in the $RUN command. Description: To copy one file, the file being copied from should be specified in the PAR= field of the $RUN command, and the file to be copied into should be attached to logical I/O unit SPUNCH. The "from" file should be specified by putting the 4 character signon ID followed by a colon(:) before the file name. For example, to copy HISFILE under the ID XXXX into -MYCOPY use the command $RUN *COPY SPUNCH=-MYCOPY PAR=XXXX:HISFILE An error comment will be produced if access to the other user's file is not permitted. To copy several files, the PAR= field should be omitted. The program will request, via GUSER, two lines for each copy: the first is the name of the "to" file and the second is the name of the "from" file. As in *PERMIT, the program will read until an end-of-file is reached. The "from" file, if it is a line file, may specify line number ranges, as for example, XXXX:HISFILE(1000,5000) but may not specify explicit concatenation. Any implicit conca- tenation specified in the "from" file will be ignored. Line files may be copied into sequential or sequential with line number files and vice-versa. It should be noted that line number ranges and the @I modifier on the "to" file operate exactly as they would on an MTS $COPY command. Presently *COPY is the only way files permitted to ALL or PRJNO may be copied. Files permitted RO may be copied either by $COPY or *COPY. A summary and more questions about stage 2 from JeffFrom: Jeff OgdenSubject: Re: file sharing in early versions of MTS Date: May 24, 2014 11:15:11 PM EDT To: Mike Alexander Cc: Gary Pirkola Thanks Mike. This is interesting. I vaguely remembered *PERMIT and RO files. I didn't know about or forgot about *COPY. I haven't looked at the source yet, but as far as managing simultaneous access to shared files, it seems as if there were two approaches: 1) Make the files RO to everyone including the owner so that the issue of simultaneous changes to files that others have open cannot occur or at least not unless you un-permit the file using *PERMIT and in that case we just cross our fingers and hope that things work out. And at least for fairly small files where the entire file will fit into a task's file buffers, things should work out OK. 2) If the files are permitted, but not RO, then let *COPY worry about the simultaneous access issues. I'm going down this path partly because I'm interested and partly because the "MTS System Architecture" article in Wikipedia doesn't currently talk about shared files or file locking and so I'm gathering information for an addition to that article. -Jeff A reply about simultaneous access during stage 2 from MikeOn May 25, 2014, at 1:05 AM, Mike Alexander wrote:It's not quite that good. *COPY didn't worry about simultaneous access issues, it just copied the file and hoped it happened quickly enough that it got something consistent. At least the original file couldn't be damaged by this, even if the copy was corrupted. In practice I never heard of a case in which this caused problems. Mike |
1-10 of 10
