New features with AN-2020-09-04:

This is the first localization step for the schily source consolidation. Many
programs now (hopefully) call gettext() for all strings that need localization.

-	The next step will include dgettext() calls for the libraries and the
	missing programs

-	The following step will include the extracted strings

-	The last step will include German translations and install support
	for the resulting binary message object files.

----------> Please test and report compilation problems! <---------

***** NOTE: As mentioned since 2004, frontends to the tools should *****
*****		call all programs in the "C" locale		   *****
*****		by e.g. calling: LC_ALL=C cdrecord ....		   *****
*****		unless these frontends support localized strings   *****
*****		used by the cdrtools with NLS support.		   *****

		*** WARNING        ***
		*** Need new smake ***

	*** Due to the fact that schily-tools 2014-04-03 introduced to use new macro
	*** expansions and a related bug fix in smake, you need a newer smake
	*** to compile this source. If your smake is too old and aborts, ensure to
	*** use the recent smake by calling:

	cd ./psmake
	./MAKE-all
	cd ..
	psmake/smake
	psmake/smake install

	The new smake version mentioned above is smake-1.2.4
	The recent smake version is smake-1.3

	*** Due to the fact that schily-tools 2018-01-26 introduced
	*** optimizations for the Schily version of SunPro Make, you
	*** need at least the dmake version from 2018/01/11 with support
	*** for the "export" directive to compile with this makefile system.


	WARNING: the new version of the isoinfo program makes use of the 
		*at() series of functions that have been introduced by Sun
		in August 2001 and added to POSIX.1-2008. For older platforms,
		libschily now includes emulations for these functions but
		these emulations have not yet been tested thoroughly.
		Please report problems!

	BUG WARNING: Please never report bugs only to Linux distributions as
		they usually do not forward these bug reports upstream and as
		the Linux distributions typically do not let skilled people
		check the bugs. We did not hear about a FIFO problem in star
		for a long time. Then a problem on Linux occurred once
		every 6000-10000 tries but it did not happen on Solaris after
		even 10 million tries, so it was not known besides Linux.

	BUG WARNING: *** GNU make *** starts too early with parallel
		execution (when reading Makefiles and evaluating rules for
		"include" statements already). Since GNU make does not
		support a concept for a correct ordering of such actions,
		you need to be prepared to see gmake fail in parallel
		mode. If you are interested in reliable parallel execution,
		it is recommended to use the included "dmake" program with
		a command line like:

			dmake -j10 -f SMakefile

		from the top level directory. Note that if you are on Linux,
		you need a halfway recent kernel or the compile time will not
		go down because of the low POSIX semaphore performance in
		older Linux kernels.

		The "dmake" program included in the schilytools tarball is the
		current version of the "new" SunOS make program that has been
		introduced in January 1986 by Sun Microsystems. It also
		introduced new features like the "include" directive that
		3 years later have been copied by gmake in a partially buggy
		way. As gmake does not fix showstopper bugs, it cannot be
		supported. Current showstoppers are: 1) gmake executes
		"include" related rules in the inverse order, causing rules
		to fail if they depend on files created by an "earlier" action
		2) gmake caches an outdated state of the directory and aborts
		with a wrong complain about allegedly missing files that in
		fact exist already.


-	Makefile system: RULES/rules.env The environment variables FIGNORE, 
	LD_LIBRARY_PATH LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64 are now
	unexported from the enviroment.

	In special FIGNORE is dangerous, as it is frequently used by bash
	users but tells ksh93 to modify it's behavior with "echo *" and
	this may cause strange things with our makefiles in case that
	/bin/sh is ksh92. This applies e.g. to Oracle Solaris 11 and
	OpenSolaris.

-	libfind: version is now 1.8

-	smake: new version date, this had been forgotten with the last change

-	cpp: new version date, this had been forgotten with the last change

-	hdump/od: added a new hardlink xpg4/bin/od to "od" for 100% POSIX
	compliant behavior. The default behavior of "od" is designed to be 
	SVID-3 compatible, since the basic "od" personality of "hdump" has
	been written to replace the closed source "od" from OpenSolaris.

-	Bourne Shell: added $(RM) -f $(DEST_DIR)$(INSBASE)/xpg4/bin/bosh
	to the Makefile in the commands before creating the symlink 
	xpg4/bin/bosh to bosh to permit to call "make install" more than
	once without causing an error.

-	Bourne Shell: version.h now contains a new version date.

-	OpenCSW compile Makefile Mocsw: Set new marker OPENCSW_MODE64=32
	in all pkginfo files.

-	OpenCSW compile Makefile Mocsw: clear RUNPATH as long as we do not
	deliver own shared libraries.

-	OpenCSW packages: many entries in the files checkpkg_override
	for the various packages have been modified to match recent rules.

-	OpenCSW compile Makefile Mcsw: The make macro INS_RBASE is now
	set to /opt/csw in order to have configuration files inside the
	csw tree.

-	CSWcdrtools package: added etc/default/cdrecord and etc/default/rscsi

-	CSWsccs package: added the new man pages to the list of files.

-	CSWschilyutils package: added "patch" to the list of programs
	in that package.

-	CSWschilyutils package: added the xpg4/bin/sh and xpg4/bin/bosh
	links.

-	CSWschilyutils package: added etc/sformat.dat

-	CSWschilyutils package: CSWschilyutils/prototype no longer lists
	ved/dotfiles.tar.gz since this file belongs into CSWved

-	CSWschilybase package: added the man pages starthandlecond.3
	and unhandlecond.3

-	CSWstar package: added share/doc/star/testscripts/pax-rename.tar.gz

-	CSWstar package: added etc/default/star and etc/default/rmt

-	CSWmake package: now includes a license file that is compatible 
	to the csw rules.

-	CSWmake package: the NAME entry has been enhanced.

-	CSWsccs package: now includes "ccs/bin/fsdiff" (see below).

-	CSWsccs package: the NAME entry has been enhanced.

-	sformat: the file sformat.dat is now also searched for in
	/opt/csw/etc/

-	SunPro Make: libmksh now compiles even if ESTALE, NOFILE, O_DSYNC or
	ETXTBSY are undefined.

	Thanks to a hint from Albert Wik.

-	autoconf: congig.guess: FreeBSD on 64 bit ARM returns arm64 from
	uname -m; this was previously not supported and rejected by config.sub.
	We now convert "arm64" into "aarch64" in config.guess to get the usual
	expected results.

-	udiff: udiff is now using getdelim() and thus supports to compare
	files with arbitrary line length.

-	udiff: The code now supports to correctly diff files with nul bytes
	inside lines. This is done by using cmpbytes() instead of streql() and
	by honoring the real line length (from getdelim) while computing the 
	line hash.

	As a result, "udiff" by default behaves similar to "diff -a" as
	implemented in newer diff(1) versions.

-	udiff: A warning is now printed if one or both files do not end in a
	newline. This is to make "udiff", when called as "fdiff" to become
	100% compatible to a POSIX "diff file1 file2".

-	udiff: The above changes, as a side effect, also result in a
	performance win with a factor of approx. 3.5x.

-	udiff: The man page has been enhanced.

-	udiff: "make install" now installs a symlink named "fdiff". If "udiff"
	is called as "fdiff", then the output is written in -posix diff format
	and the "nmatch" parameter is set to a value of 1 in order to give
	best POSIX diff(1) compatibility.

-	udiff: "make install" now installs a symlink named "fsdiff". If "udiff"
	is called as "fsdiff", then the exit code is compatible to bdiff(1) and
	this allows it to be used as the "diff" program for SCCS delta(1).

	Exit code compatibility to bdiff(1) means that the exit code is 
	only != 0 in case of an error, but not because of a difference between
	both files.

-	udiff: the CSWschilyutils package now includes this fdiff/fsdiff symlinks

-	udiff: the CSWschilyutils package now includes this fdiff.1/fsdiff.1 
	man pages.

-	patch: The random unit tests gentest and cmptest now add "...;exit" to
	the trap command in order to let it work correctly when aborted by a
	signal.

-	SCCS: The random unit tests gentest and cmptest now add "...;exit" to
	the trap command in order to let it work correctly when aborted by a
	signal.

-	SCCS: The unit tests now work with gmake and dmake as well.

	This did not work anymore with the last version of schilytools, after
	adding support that allows to put the SCCS test code/data on a dedicated
	"tmpfs" filesystem has been added to speed up tests on Linux as a work
	around for the slow filesystems on Linux.

-	SCCS: defines.h now includes #ifdef NO_MMAP to disable the use of mmap()
	in SCCS.

-	SCCS: get(1) and delta(1) now restore the removed newline at the end of
	the last line if this was a SCCSv6 "^ANxxx" line to support files not
	ending in a newline. This is needed since we now use mmap() to read the
	file and a second "read" would no longer automatically recover the
	original data.

-	SCCS: The delta(1) command now correctly handles deltas where the last
	difference hits the end of the history file. This is a problem that was
	introduced while converting SCCS to use mmap() a month ago. In theory,
	this bug should have been detected by the Unit Test Suite, but this did
	not happen on Linux since the input data in that specific case was the
	concatenation of all /bin/sh* files. Running the test suite on
	Solaris detected the problems since the files on Solaris has a different
	size and that triggered the bug:

	Linux did use input data that does not cause the EOF situation in the 
	mmap()ped area that was needed to trigger the bug. This caused the EOF
	recognition to hit later in the final readmod() loop that is intended
	to copy the residual trail from the history file.

	On Solaris, with the data in use, EOF hits earlier and since EOF now 
	causesd grewind() to be called earlier, the final readmod() loop may
	have been run even though there was no data left over. This did currupt
	the history file. We now no longer call the final readmod() loop if
	grewind has already been called, since that signals that the whole
	s.inputfile has already been consumed.

	So our bug did depend on the size and data from the file to be checked
	in.

	When EOF hits earlier, this did also cause the remembered checksum to
	be remembered to early. We now correct the remembered checksum value 
	if there are later calls to putline().

-	SCCS: The delta(1) command now removes the e.file in case of an error
	abort. The e.file is a temporary file, created for uuencoded data and
	previous versions of the delta command did leave this temporary file
	laying around in case of an error.

-	SCCS: admin(1) now avoids a core dump in case that the internal
	variable dir_name is a NULL pointer.

-	SCCS: The "udiff" command is now installed as /opt/schily/ccs/bin/fsdiff
	as well, to allow it to be used as a "bdiff" alternative by
	SCCS delta(1).

	"udiff" is faster than "bdiff" and does not need to diff the files
	in a segmented mode, since "udiff" does not slow down with large files.
	The segmentation from "bdiff" (bdiff calls diff(1) with parts of the
	file) is needed in order to avoid a too big slow down with larger files.
	This however causes a larger diff output than what would bee created
	with a single "diff" call. Using "udiff" for SCCS delta(1) thus is able
	to reduce the size of the SCCS history files.

-	SCCS: The delta(1) command now may be configured to use "fsdiff" as
	default diff program by compiling delta(1) with -DUSE_FSDIFF

	In that case, "delta -b ..." allows to tell delta(1) tu use "bdiff"
	as before.

	The switch to use fsdiff by default will be made later after more
	in depth testing has been done. We currently "only" have approx. one
	million successful tests with a delta(1) that uses "fsdiff".

-	SCCS: The script rcs2sccs now adds "...;exit" to the
	trap command in order to let it work correctly when aborted by a signal.

-	SCCS: the -X option no longer sets a flag that indicates that an option
	with a string parameter is present when the related string parameter is
	of zero length. Since all "-Xname=value" type options currently make no
	sense if used with an empty string like "-Xname=", this is a useful
	change to avoid core dumps from trying to access NULL pointers (as it
	has been handled by the previous version) in the various programs.

-	SCCS unit tests: several delivered SCCS history files for the unit
	tests have been converted to no longer need a uuencoded SCCS history
	since SCCSv6 supports to handle files that have lines starting with ^A
	or a last line in a file not ending with a newline.

-	SCCS: diff(1): the comment has been enhanced. Background is a
	potentially incorrect bugfix from Sun Microsystem from the past. The
	related change is supposed to fix a problem with diff3: It is
	claimed to silently mismerge files. The visible effect of that change
	is that diff(1) no longer always finds the longest identical
	subsequence following a change.

	For now, we still leave the code as is even though we believe it is
	wrong but the current code allows the diff(1) output to be compared
	with the output from "udiff -posix nmatch=1 ..." in nearly all cases
	and this is a benefit for debugging.

-	SCCS: The recent FreeBSD linker no longer supports COMMON variables.
	We now define the variable xtimezone only in files that contain
	a main() entry and extern long xtimezone in all other files.

	It would be a nice idea if the FreeBSD linker did just flag COMMON
	variables with the same name but different size as this is done by
	the Solaris linker.

	Thanks to Nico Sonack for reporting

-	SCCS: a new version date has been set up to identify the new version
	of SCCS.



-	SCCS: The current idea for converting a historic SCCS project into
	a project oriented SCCS history bundle is the following:

	-	Create a user map file for "sccslog" by calling:

		mkdir $HOME/.sccs
		$EDITOR $HOME/.sccs/usermap

		Enter the UNIX login names followed by a TAB, followed
		by an E-mail notation. Use one line per user, e.g.

			joerg	J. Schilling <joerg@mail.com>

	-	Create a copy of the whole project to work on for this test.
		Do not do this conversion on the original project until
		sccs-6.0 is ready.

	-	chdir to the project home directory of the just created copy.

	-	Call "sccs init -i ." to make the project using an in-tree
		project oriented repository.

	-	Call:

		find * -path '*SCCS/s.*' | /opt/schily/ccs/bin/sccscvt -NSCCS/s. -k -ooo -V6 -

		for the CSRG BSD project use:

		find * -path '*SCCS/s.*' | TZ=US/Pacific /opt/schily/ccs/bin/sccscvt -NSCCS/s. -k -ooo -V6 -

		to convert all history files into SCCSv6 history files. The
		TZ=US/Pacific is important for the UCB conversion since SCCSv6
		uses timezones but SCCSv4 does not and we need to have the 
		correct timezone entries in the SCCSv6 history files.

		For the complete "schilytools" project with 4200 SCCS history
		files in 55 Mbytes, this takes 12 seconds for the SCCS history
		from 1984 .. 2020, but note that most of the edits from the
		1980s are lost, so there are few entries from the time
		before 1989.

		An alternate example: the SCCS history from the BSD-4.4 project
		from December 1979 up to June 1995 is in 12600 SCCS history
		files that take up 125 MB.
		The conversion time to the SCCSv6 history file format is
		18 seconds.

	-	Call:

		find * -path '*SCCS/s.*' | /opt/schily/ccs/bin/sccslog -changeset -

		to populate the changeset file from the existing deltas.

		For the complete "schilytools" project with 19600 commits,
		this takes 8 minutes. The resulting file .sccs/SCCS/s.changeset
		has a size of approx. 7 MBytes.

		An alternate example: the SCCS history from the BSD-4.4 project
		from December 1979 up to June 1995 has approx. 47000 commits.
		The conversion time is approx. 40 minutes.
		The size of the resulting changeset file is aprox. 14 MBytes.

	-	convert the in-tree repository into an off-tree repository.
		This final step is not yet needed and there is currently no
		code to do that automatically.

	-	If you like to check the resulting changeset file, there is
		currently only one way to look at it, by calling:

		sccs -O get -p -A -m .sccs/SCCS/s.changset | more

		This prints an annotated version of the changeset file.
		The next task is to develop an enhancement to "sccs log"
		that prints the changeset in a way similar to what "hg log -v"
		prints.

	-	NOTE: Normal filesystems on Linux are slow, it is advised to
		make the conversions on tmpfs for performance reasons in case
		you are using Linux.

	Please however keep in mind that this is still experimental and there is
	absolutely no grant that a changelog created with current experimental
	software will work correctly with the final SCCS version. The procedure
	is just an example to check how it may look like.

	The final conversion method will be more automated... most likely
	by a command similar to "sccs import ..."

	IMPORTANT: This is not yet the time to finally convert a project into
	the project mode, because the project would be stuck in the current 
	state. What we need to continue work in that repository state in the
	project mode is at least a working "sccs commit". Be prepared to remove
	the changeset history file once "sccs commit" works and to re-create
	the changeset file for that time.

 

-	SCCS TODO:

	-	Activate "fsdiff" as a "bdiff" replacement in delta(1)
		to speed up delta(1) and to reduce the size of the SCCS
		history files.

	-	Implement something that outputs similar information from
		the changeset file as printed with "hg log -v".

		This would be the next key feature.

	-	verify whether sccs.c uses -NSCCS in the back end programs
		correctly, instead of converting g-file names from the command
		line into s.file names in the frontend in order to forward 
		s.file names to the backend programs. This is neded for an
		off-tree repository.

		The related unit tests are already passed.

	-	Add code to to sccs(1) to send a list of files to admin(1) and
		delta(1) with new or modified files in order to have all
		important code for a "sccs commit" in a single program that
		does not need to deal with ARG_MAX limitations.

	-	Add code to admin(1), delta(1), sccs-log(1) and get(1) to 
		maintain/understand the changeset file.

		This is mainly writing out the sccschangeset(4) entries to an
		intermediate store if a single file has been treated
		successfully. For sccs-log(1), see below.

	-	Finish the work to allow normal line based diffs in SCCS even
		for binary files. This are files that include nul bytes and
		this needs to completely avoid fputs() and this needs an 
		initialized member p_line_length in struct packet even for 
		all content that does not result from a previous getline() call.

	-	sccs -R tell (and probably other subcommands?) does not yet
		work in NewMode

	-	Add code to libcomobj to understand the changeset file.
		This is needed in order to e.g. know the file names and file
		specific SIDs/state that corresponds to a project global SID.

	-	Find/verify a complete transactional model that allows to repair
		complex changes to the set of files for a project that have
		been aborted in the middle. The current idea is to create the
		file $PROJECTHOME/.sccs/changeset with the deltas to the
		changeset during a complex update operation.

	-	Find a decision on how to deal with the admin flags that are
		currently implemented as global flags and thus do not depend on
		the SID (version) if the history file.

	-	Aborting a transaction via ^C currently requires a manual
		removal of the global lock file. Find a way to avoid this in
		case that a commit has been aborted while being prompted for
		a commit message (which is before any real action happened).

	-	Implement a fully automated method to convert a SCCSv4 based
		history with unrelated history files into a new SCCSv6 based
		project mode history with a populated changeset history file.

		This will most likely be done as a variant of the to be defined
		new command "sccs sccsimport" that imports a whole existing old
		SCCS project.

	-	Implement this "sccs sccsimport" based conversion in a way where
		sccs(1) holds the global changeset lock for the whole time
		of the conversion.




-	Bourne Shell Missing features for POSIX compliance:

	- Support for $'...' quoting (this is not needed for the current
					version of POSIX but for the next POSIX
					version that will be named SUSv8).
					The development of SUSv8 will start in
					late 2016.

	We are now expecting the Bourne Shell to be fully POSIX compliant.

-	Bourne Shell further TODO list:

	-	Finish loadable builtin support.

	-	POSIX does not allow us to implement ". -h", so we will
		add a "source" builtin to be able to implement "source -h"

-	The following builtins (that are available in bsh) are still missing in
	the Bourne Shell:

	err			echo with output going to stderr
	glob			echo with '\0' instead of ' ' between args
	env			a builtin version of /usr/bin/env

	The following bsh intrinsics are still missing in the Bourne Shell:

	-			the restricted bsh has restriction features that
				are missing in the Bourne shell.

	-	source -h	read file into history but do not execute

	and probably more features not yet identified to be bsh unique.



Author:

Joerg Schilling
D-13353 Berlin
Germany

Email: 	joerg@schily.net, joerg.schilling@fokus.fraunhofer.de

Please mail bugs and suggestions to me.
