New features with AN-2020-07-18:

This is the first localization step for the schily source consolidation. Many
programs now (hopefully) call gettext() for all strings that need localization.

-	The next step will include dgettext() calls for the libraries and the
	missing programs

-	The following step will include the extracted strings

-	The last step will include German translations and install support
	for the resulting binary message object files.

----------> Please test and report compilation problems! <---------

***** NOTE: As mentioned since 2004, frontends to the tools should *****
*****		call all programs in the "C" locale		   *****
*****		by e.g. calling: LC_ALL=C cdrecord ....		   *****
*****		unless these frontends support localized strings   *****
*****		used by the cdrtools with NLS support.		   *****

		*** WARNING        ***
		*** Need new smake ***

	*** Due to the fact that schily-tools 2014-04-03 introduced to use new macro
	*** expansions and a related bug fix in smake, you need a newer smake
	*** to compile this source. If your smake is too old and aborts, ensure to
	*** use the recent smake by calling:

	cd ./psmake
	./MAKE-all
	cd ..
	psmake/smake
	psmake/smake install

	The new smake version mentioned above is smake-1.2.4
	The recent smake version is smake-1.3

	*** Due to the fact that schily-tools 2018-01-26 introduced
	*** optimizations for the Schily version of SunPro Make, you
	*** need at least the dmake version from 2018/01/11 with support
	*** for the "export" directive to compile with this makefile system.


	WARNING: the new version of the isoinfo program makes use of the 
		*at() series of functions that have been introduced by Sun
		in August 2001 and added to POSIX.1-2008. For older platforms,
		libschily now includes emulations for these functions but
		these emulations have not yet been tested thoroughly.
		Please report problems!

	BUG WARNING: Please never report bugs only to Linux distributions as
		they usually do not forward these bug reports upstream and as
		the Linux distributions typically do not let skilled people
		check the bugs. We did not hear about a FIFO problem in star
		for a long time. Then a problem on Linux occurred once
		every 6000-10000 tries but it did not happen on Solaris after
		even 10 million tries, so it was not known besides Linux.

	BUG WARNING: *** GNU make *** starts too early with parallel
		execution (when reading Makefiles and evaluating rules for
		"include" statements already). Since GNU make does not
		support a concept for a correct ordering of such actions,
		you need to be prepared to see gmake fail in parallel
		mode. If you are interested in reliable parallel execution,
		it is recommended to use the included "dmake" program with
		a command line like:

			dmake -j10 -f SMakefile

		from the top level directory. Note that if you are on Linux,
		you need a halfway recent kernel or the compile time will not
		go down because of the low POSIX semaphore performance in
		older Linux kernels.

		The "dmake" program included in the schilytools tarball is the
		current version of the "new" SunOS make program that has been
		introduced in January 1986 by Sun Microsystems. It also
		introduced new features like the "include" directive that
		3 years later have been copied by gmake in a partially buggy
		way. As gmake does not fix showstopper bugs, it cannot be
		supported. Current showstoppers are: 1) gmake executes
		"include" related rules in the inverse order, causing rules
		to fail if they depend on files created by an "earlier" action
		2) gmake caches an outdated state of the directory and aborts
		with a wrong complain about allegedly missing files that in
		fact exist already.



-	Bourne Shell: A new symlink from /opt/schily/xpg4/bin/bosh to
	/opt/schily/xpg4/bin/sh is created when "smake install" is called.
	This helps people to call "bosh" in strict POSIX mode using the
	command line name "bosh" by putting /opt/schily/xpg4/bin in PATH
	before /opt/schily/bin, where the bosh implements better backwards
	cmpatibility to the Bourne Shell by default than a strictly POSIX
	compliant bosh would do. /opt/schily/xpg4/bin/bosh behaves the same
	as "/opt/schily/bin/bosh -o posix".

	Thanks to  Koichi Nakashima for reporting.

-	Makefile system: changed several "echo MAKING... && sh ./MKLINKS"
	into "echo MAKING... ; sh ./MKLINKS"

	This allows an optimization in smake to avoid a call to sh -c "cmd" 
	if the command is a simple command or a simple command prefixed by
	"echo something ;". Since 2012, we implement an inline echo command
	in smake to speed up makefile execution.


-	smake: The -a option is now no longer missing in the smake -help
	output. This has been forgotten when the option has been added in March.


-	star: star could dump core if it was used as "star -t ..." or 
	"star -x ..." while being in a UTF-8 based locale and trying to deal
	with extremely long pathnames (more than PATH_MAX) in the archive.
	
	This bug was caused by the dummy conversion routines _to_utf8() or
	_from_utf8() that did not stop after "tolen" bytes (the current size
	of the dynamically growing path structure) have been copied.
	This bug has been introduced in 2018 when the dynamic path name
	variables have been introduced together with support for extremely
	long path names.

-	star: lpath_unix.c and lhash.c renamed a local variable buflen to bflen
	to avoid a gcc shadowing warning with the rest of star.

-	star: star.c fixed some fallthrough warnings from lint.

-	star: The FIFO code (which is 30 years old) did use an int for the size
	which historically was OK, but this did limit the size of the FIFO to
	2 GB. Now with modern tape drives that are really fast, a FIFO with
	2 GB would only give a tape streaming reserve for approx. 8 seconds,
	which is not sufficient. Approx. 30 seconds reserve are recommended.
	With modern tape drives, this would need approx. 8 GB of FIFO. Be 
	however careful not to use more than half of the real RAM available
	in the whole system for the star FIFO.

	Note that this change induced the need for a lot of derived changes
	in approx. 1000 lines of code spread over the whole star project and
	for this reason, it is advised to carefully test the new version and
	to report if problems occur.

-	fifo: the fifo command is based on the star FIFO code and has been
	changed to support a FIFO size > 2 GB as well.


-	cdda2wav: on FreeBSD, with an unsupported attempt to use a /dev/*
	based device specifier (dev=/dev/something) together with -vall,
	cdda2wav did dump core.

	We now abort cdda2wav before it could dump core and we removed the
	outdated code from the 1990s that could automatically switch the
	work mode back into "coocked_ioctl" in case we are on FreeBSD >= 6.0.
	This code does not make sense at all on newer FreeBSD releases and
	coocked ioctls are a method from the past that give really bad
	results.

	Thanks to a hint from Akos Somfai

-	cdrtools: A new version date has been introduced to allow to identify
	the fixed version.


-	SCCS: New separate man pages sccs-branch.1, sccs-clean.1,
	sccs-create.1, sccs-diffs.1,sccs-enter.1 sccs-fix.1, sccs-ldiffs.1,
	sccs-print.1 and sccs-unget.1 have been created.

-	SCCS: New preliminary man pages sccs-add.1, sccs-commit.1,
	sccs-remove.1 sccs-rename.1 and sccs-status.1 have been created.

-	SCCS: We now have separate man page for every SCCS command and for
	every sccs(1) subcommand.

-	SCCS: We now have 41 separate man pages for SCCS commands.

-	SCCS: A wrong double space character in the "cmds" help file has been 
	removed.

-	SCCS: The "sccs log" command has been enhanced to remove the
	nanoseconds from the "date and time created ..." message text, to be
	able to bundle several "sccs create" commands into a single message
	even when operating on SCCSv6 history files.

-	SCCS: The locking method has been made more robust against long running
	programs. The challenge is a program on a different NFS connected
	computer and long delays, e.g. via a conguestion on the Wlan that is
	used for the NFS connection. We do this by installing a SIGALRM 
	controlled timer that refreshes the lock file every every 30 seconds.

	Up to two lockfiles are refreshed with this method on a regular base:

	-	The lock for the file currently worked on

	-	The global project lock.

	Note that this only works on a POSIX system with SA_RESTART used 
	with sigaction(), or on a BSD system with sigvec() as otherwise
	the signal would interrupt read() and write() calls.

	Hint: A typical operation for SCCS in the historic behavior was
	related to approx. 100kB of data, while when in the new project mode,
	we need to be prepared to handle more than 1 GB with an "atomic"
	operation. This is why we need to make locking more robust against
	long lasting operations.

	Hint (see below): the conversion of the BSD-4.4 SCCS history from
	a single file based SCCSv4 history to a project oriented SCCSv6
	history causes almost 400 GB of changes to be written, even though
	the maximum single file write is only 14 MB.

-	SCCS: libmpw:lockit.c has been made more robust against NULL pointer
	filenames.

-	SCCS: libmpw:lockit.c now detects whether a locking attempt is made
	that would lead to a deadlock situation because the file is already
	locked by the same process or by it's parent. Lockit() now immediately
	returns with an error code instead of waiting 2 minutes before
	giving up.

-	SCCS: libmpw:lockit.c A new generic function lockfatal() prints
	locking error messages has been created. This function also prints
	messages that point to dead lock situations.

-	SCCS: new error codes cm23 and cm24 have been created that contain
	help for deadlock situations.

-	SCCS: libcomobj: auxf.c and lockchset.c have been made more robust 
	against NULL pointer filenames.

-	SCCS: libcomobj: A new function islockchset() has been introduced in
	order to check whether a lock file is the same file as an existing
	lock file for the project global changset file. Since the lock file
	for the changeset file is used as the global lock and since it would
	cause a deadlock, if a SCCS command did try to create a file specific
	lock file for the same file again while trying to modify the current
	version of the changeset file.

-	SCCS: the commands admin delta get cdc/rmdel sccscvt and unget now all
	call lockit() for the file under work only, if islockchset() returns
	FALSE. This is needed in order to avoid deadlocks in case that the
	file, that is worked on, is the changeset file and we also need to hold
	a global lock because we are in project mode.

	Without that new code, we would first create a lock to the changeset
	file as the global lock and then a second lock to the changeset file 
	as it is the current file to be processed. Two locks attempts to the 
	same file would result in a deadlock situation that was not handled
	before.

-	SCCS: libcomobj: date_ab.c now supports a new function parse_datez()
	that allows to specify a datetime record with nanoseconds and with
	a timezone. This is used for the new option -Xdate=datetime.

-	SCCS: libcomobj/bulk.c: a leading "./" is stripped off the input
	filename for bulkprepare() in order to get reliable and normalized
	path names for the "initial_path" property of SCCSv6 history files.

-	SCCS: libcomobj: parsex.c now correctly parses a -X option like:

		-Xopta,optb=value

	by verifying that a '=' is only honored in case there is not a ','
	before a '=' in the -X option list.

-	SCCS: The commands admin(1) and delta(1) now support new options:

	-Xnobulk	To disable -N (bulkmode) filename translations.
			This is needed to disable a -N option that is 
			automatically added by sccs(1) in NewMode and that
			would otherwise make it impossible to deal with the
			changeset file ".sccs/SCCS/s.changeset" that is
			outside the naming rules for normal files.

	-Xdate=datetime	To overwrite the usual methods to determine the
			time stamp used for a new delta. This is needed
			to e.g. convert a historic SCCS history into a
			new project oriented bundle history.

			In future, it could be used to speed up RCS/SCCS
			conversion.

-	SCCS: the admin.1 and delta.1 man pages now mention -Xnobulk
	and -Xdate=datetime

-	SCCS: The command delta(1) now supports a new option:

	-Xgpath=g-path	Specify a different path to the g-file instead
			of deriving the path from the s-file using an
			algorithm that may not apply in a specific case. 
			This option is needed to manage the changeset file,
			in special when converting a historic SCCS history
			into a new project oriented bundle history.

-	SCCS: all programs that support the -X option: This option is
	now allowed to be specified more than once on the command line.
	This helps to avoid comma separated lists in favor of separate
	sub-options.

-	SCCS: various man pages: the -X option has been added to the SYNOPSIS
	section.

-	SCCS: the delta.1 man page now mentions -Xgpath=g-path

-	SCCS the command "sccs" now supports new global options:

	-N	Enforce new mode

	-O	Enforce Old Mode (the mode from Eric Allman from 1980)

	This makes it easier to use the "other" mode, and to disregard
	the current default.

-	SCCS: the sccs.1 man page and the "sccs help sccs" output have
	been enhanced to mention the -N/-O options.

-	SCCS: The command delta(1) now closes the opened file
	.sccs/changelog if it detects that the file .sccs/SCCS/s.changeset
	is the file that is just updated.

	This is needed in order to avoid that we output changeset records
	to that file. Such an action would destroy the current changeset
	collection while we add it to the changeset history file.

-	SCCS: "sccs init" could (with the changes from schily-2020-07-01)
	incorrectly create a file .sccs/SCCS/SCCS/s.s.changeset instead of
	.sccs/SCCS/s.changeset. This is now fixed by using the option:

		-Xunlink,nobulk,Gp=.sccs/changeset

	instead of just -Xunlink.

-	SCCS: "sccs log" now reads the urandom number and the initial
	path from SCCSv6 history files. This is needed in order to
	create a changeset file while converting a historic single file
	repository into a new project mode repository.

-	SCCS: "sccs log" now reads and remembers nanosecond based 
	timestamps. This is needed in order to create a SCCSv6 changeset
	file with the right timestamps.
 
-	SCCS: "sccs log" now supports a new option -R/-reverse to invert
	the sort order of the output. This prints older entries first.

-	SCCS: "sccs log" now supports a (currently) experimental option
	-changeset to create changeset lists from a single file group
	repository. The option -changeset automatically enables
	-reverse as well. This is the first step for automatically
	converting an old type SCCS repository that just holds a list of
	files managed spearately into a new project oriented repository
	with a changeset history file.

-	SCCS: "sccs log" is now able to deal with inverse timestamp order
	while creating delta groups.

-	SCCS: "sccs log" no longer allows delta groups that ovelap when
	called with the -changeset option. This is important in order to
	be able to create project changesets with a useful overall time order.

-	SCCS: "sccs log" no longer allows delta groups to contain different
	committing users when called with the -changeset option.

-	SCCS: "sccs log" no longer reads the whole content of SCCS history
	files but stops after reading the meta data. This speeds up the
	"sccs log" command by a typical factor of 2.

-	SCCS: "sccs log -changeset ..." is now able to simulate a
	"sccs commit" that has been done in the past.

	This creates (populates) the file $PROJECT_HOME/.sccs/SCCS/s.changeset
	in a way as if "sccs commit" had been used in the whole past for the
	project.

-	SCCS: "sccs log -changeset ..." now uses the programmer name from the
	delta as the programmer name for changeset entry.

-	SCCS: The version date has been updated


-	SCCS: The current idea for converting a historic SCCS project into
	a project oriented SCCS history bundle is the following:

	-	Create a user map file for "sccslog" by calling:

		mkdir $HOME/.sccs
		$EDITOR $HOME/.sccs/usermap

		Enter the UNIX login names followed by a TAB, followed
		by an E-mail notation. Use one line per user, e.g.

			joerg	J. Schilling <joerg@mail.com>

	-	Create a copy of the whole project to work on for this test.
		Do not do this conversion on the original project until
		sccs-6.0 is ready.

	-	chdir to the project home directory of the just created copy.

	-	Call "sccs init -i ." to make the project using an in-tree
		project oriented recpository.

	-	Call:

		find * -path '*SCCS/s.*' | /opt/schily/ccs/bin/sccscvt -NSCCS/s. -k -ooo -V6 -

		to convert all history files into SCCSv6 history files.

		For the complete "schilytools" project with 4200 SCCS history
		files in 55 Mbytes, this takes 2 minutes for the SCCS history
		from 1984 .. 2020, but note that most of the edits from the
		1980s are lost.

		An alternate example: the SCCS history from the BSD-4.4 project
		from December 1979 up to June 1995 is in 12600 SCCS history
		files that take up 125 MB.
		The conversion time to the SCCSv6 history file format is
		18 seconds.

	-	Call:

		find * -path '*SCCS/s.*' | /opt/schily/ccs/bin/sccslog -changeset -

		to populate the changeset file from the existing deltas.

		For the complete "schilytools" project with 19600 commits,
		this takes 9 minutes. The resulting file .sccs/SCCS/s.changeset
		has a size of approx. 7 MBytes.

		An alternate example: the SCCS history from the BSD-4.4 project
		from December 1979 up to June 1995 has 49500 commits.
		The conversion time is approx. 50 minutes.
		The size of the resulting changeset file is aprox. 13.5 MBytes.

	-	convert the in-tree repository into an off-tree repository.
		This final step is not yet needed and there is currently no
		code to do that automatically.

	-	If you like to check the resulting changeset file, there is
		currently only one way to look at it, by calling:

		sccs -O get -p -A -m .sccs/SCCS/s.changset | more

		This prints an anannotated version of the changeset file.
		The next task is to develop an enhancement to "sccs log"
		that prints the changeset in a way similar to what "hg log -v"
		prints.

	-	Normal filesystems on Linux are slow, it is advised to
		make the conversions on tmpfs for permformance reasons.

	Please however keep in mind that this is still experimental and there is
	absolutely no grant that a changelog created with current experimental
	software will work correctly with the final SCCS version. The procedure
	is just an example to check how it may look like.

	The final conversion method will be more automated... most likely
	by a command similar to "sccs import ..."

	IMPORTANT: This is not yet the time to finally convert a project into
	the project mode, because the project would be stuck in the current 
	state. What we need to continue work in that repository state in the
	project mode is at least a working "sccs commit". Be prepared to remove
	the changeset history file once "sccs commit" works and to re-create
	the changeset file for that time.

 

-	SCCS TODO:

	-	verify whether sccs.c uses -NSCCS in the back end programs
		correctly, instead of converting g-file names from the command
		line into s.file names in the frontend in order to forward 
		s.file names to the backend programs. This is neded for an
		off-tree repository.

		The related unit tests are already passed.

	-	Add code to to sccs(1) to send a list of files to admin(1) and
		delta(1) with new or modified files in order to have all
		important code for a "sccs commit" in a single program that
		does not need to deal with ARG_MAX limitations.

	-	Add code to admin(1), delta(1), sccs-log(1) and get(1) to 
		maintain/understand the changeset file.

		This is mainly writing out the sccschangeset(4) entries to an
		imtermediate store if a single file has been treated
		succsessfully. For sccs-log(1), see below.

	-	Implement something that outputs similar information from
		the changeset file as printed with "hg log -v".

	-	sccs -R tell (and probably other subcommands?) does not yet
		work in NewMode

	-	Add code to libcomobj to understand the changeset file.
		This is needed in order to e.g. know the file names and file
		specific SIDs/state that corresponds to a project global SID.

	-	Find/verify a complete transactional model that allows to repair
		complex changes to the set of files for a project that have
		been aborted in the middle. The current idea is to create the
		file $PROJECTHOME/.sccs/changeset with the deltas to the
		changeset during a complex update operation.

	-	Find a decision on how to deal with the admin flags that are
		currently implemented as global flags and thus do not depend on
		the SID (version) if the history file.

	-	Aborting a transaction via ^C currently requires a manual
		removal of the global lock file. Find a way to avoid this in
		case that a commit has been aborted while being prompted for
		a commit message (which is before any real action happened).

	-	Implement a fully automated method to convert a SCCSv4 based
		history with unrelated history files into a new SCCSv6 based
		project mode history with a populated changeset history file.

		This will most likely be done as a variant of the to bedefined
		new command "sccs import" that imports a whole existing old
		SCCS project.

	-	Implement this "sccs import" based conversion in a way where
		sccs(1) holds the global changeset lock for the whole time
		of the conversion.




-	Bourne Shell Missing features for POSIX compliance:

	- Support for $'...' quoting (this is not needed for the current
					version of POSIX but for the next POSIX
					version that will be named SUSv8).
					The development of SUSv8 will start in
					late 2016.

	We are now expecting the Bourne Shell to be fully POSIX compliant.

-	Bourne Shell further TODO list:

	-	Finish loadable builtin support.

	-	POSIX does not allow us to implement ". -h", so we will
		add a "source" builtin to be able to implement "source -h"

-	The following builtins (that are available in bsh) are still missing in
	the Bourne Shell:

	err			echo with output going to stderr
	glob			echo with '\0' instead of ' ' between args
	env			a builtin version of /usr/bin/env

	The following bsh intrinsics are still missing in the Bourne Shell:

	-			the restricted bsh has restriction features that
				are missing in the Bourne shell.

	-	source -h	read file into history but do not execute

	and probably more features not yet identified to be bsh unique.



Author:

Joerg Schilling
D-13353 Berlin
Germany

Email: 	joerg@schily.net, joerg.schilling@fokus.fraunhofer.de

Please mail bugs and suggestions to me.
