[foma] 01/02: Imported Upstream version 0.9.18~r248
Tino Didriksen
tinodidriksen-guest at moszumanska.debian.org
Mon Jan 11 08:40:10 UTC 2016
This is an automated email from the git hooks/post-receive script.
tinodidriksen-guest pushed a commit to branch master
in repository foma.
commit 2c209cd4cc24ba4e1958db3941622ecfe066c239
Author: Tino Didriksen <mail at tinodidriksen.com>
Date: Mon Jan 11 08:40:01 2016 +0000
Imported Upstream version 0.9.18~r248
---
CHANGELOG | 148 +++
COPYING | 202 +++
Makefile | 110 ++
README | 44 +
README.cmatrix | 19 +
README.symbols | 26 +
apply.c | 1489 +++++++++++++++++++++
cgflookup.c | 303 +++++
cmatrix.l | 97 ++
coaccessible.c | 176 +++
constructions.c | 3162 ++++++++++++++++++++++++++++++++++++++++++++
contrib/foma2js.perl | 173 +++
contrib/foma_apply_down.js | 49 +
define.c | 144 ++
determinize.c | 881 ++++++++++++
dynarray.c | 719 ++++++++++
extract.c | 71 +
flags.c | 518 ++++++++
flookup.c | 385 ++++++
foma.c | 243 ++++
foma.h | 161 +++
fomalib.h | 505 +++++++
fomalibconf.h | 305 +++++
iface.c | 1713 ++++++++++++++++++++++++
int_stack.c | 96 ++
interface.l | 655 +++++++++
io.c | 1013 ++++++++++++++
lexc.h | 11 +
lexc.l | 245 ++++
lexcread.c | 1087 +++++++++++++++
mem.c | 95 ++
minimize.c | 673 ++++++++++
python/foma.py | 495 +++++++
regex.l | 461 +++++++
regex.y | 420 ++++++
reverse.c | 48 +
rewrite.c | 610 +++++++++
sigma.c | 422 ++++++
spelling.c | 880 ++++++++++++
stack.c | 221 ++++
stringhash.c | 101 ++
structures.c | 905 +++++++++++++
topsort.c | 170 +++
trie.c | 151 +++
utf8.c | 272 ++++
45 files changed, 20674 insertions(+)
diff --git a/CHANGELOG b/CHANGELOG
new file mode 100644
index 0000000..2468a36
--- /dev/null
+++ b/CHANGELOG
@@ -0,0 +1,148 @@
+0.9.18alpha (20150612)
+
+- Many bugfixes and speedups
+- Corrects handling of some rare unicode composing diacritics
+- Python bindings
+- Adds _closeu()-builtin function which closes sigma
+- Adds _sublabel(L1,symbol,L2)-builtin function: substitute all instances of symbol in L1 with L2
+- Adds separate pairs, pairs > file, random-pairs commands in interface
+- Adds possibility to automatically align lexc-entries
+
+0.9.17alpha (20121117)
+
+- Many bugfixes in foma, flookup (apply code)
+- Adds possibility to redirect "print words" output to file by "print words > file"
+
+0.9.16alpha (20111213)
+
+- New faster apply code, as well as optional indexing of arcs in flookup
+- adds rewrite rule formalism "transducers with backreferences" (e.g. T -> || L _ R, where T is a transducer)
+- flookup now has option to run as UDP server (-S)
+- Adds low-level functions _marktail(), _addfinalloop(), _addnonfinalloop(), _addsink(), leftrewr(), _flatten().
+- Minor bugfixes in apply code
+- Some rare memory leak fixes throughout
+- Adds functions so that apply_med() can be used through the API
+- Adds functions to API fsm_get_next_state(), fsm_get_next_state_arc()
+
+0.9.15alpha (20111110)
+
+- adds cgflookup utility to facilitate piping with Constraint Grammar parsers
+- flookup now applies in chain (a virtual compose) if there are several transducers in a file, except if the [-a] flag is given, in which case flookup looks for an output in each transducer until it finds one (simulating priority union .P.)
+- changes and bug fixes in application code/flag diacritic handling
+- lexc now accepts forced alignments with zeroes and info strings
+- \r (carriage return) awarness added
+- adds support for the construct regex @re"regexfile.txt" (assumes file has one regular expression on one line)
+- fixes argument order of interface commands compose net and concatenate net
+- adds functions in API to read in multiple nets from one file through an iterator fsm_read_binary_file_multiple_init(infilename)/fsm_read_binary_file_multiple(rh))
+- improved error messages
+- adds "test sequential" ("tseq" for short) and assert stack commands
+- fixes bugs with loading/saving nets that have newlines in transitions
+
+0.9.14alpha (20110203)
+
+- Changed tokenization in apply_up() and apply_down() to always choose the leftmost longest tokenization in case of multicharacter symbols sharing the same prefix. Previously, all possible tokenizations were given. For example, earlier regex [a a|aa] would match the input string "aa" in two ways, yielding two outputs, whereas only one tokenization is given now, the leftmost-longest one. This also speeds up the apply functions.
+- Fix minor bug in fsm_construct_set_final() and fsm_construct_set_initial(), affecting fsm_reverse()
+- Minimize automata read with "read text" and "read spaced-text"
+- Minor change in .dot output style
+- Fix bug in fsm_trie
+
+0.9.13alpha (20101025)
+
+- Various memory-management improvements throughout
+- Adds the commands "read text" and "read spaced-text" for building automata/transducers from word lists. The functionality can be embedded in regexes with @txt"filename" and @stxt"filename". From the API, the functions fsm_read_text_file() and fsm_read_spaced_text_file(), reads files and builds trie FSMs from the lists
+- Adds the external utility flookup, which reads words from stdin and applies them to a transducer given in a file, and prints output to stdout
+- Adds the command "substitute defined", and the corresponding fsm_substitute_label() in the API.
+- Improved prolog file reading and writing
+- Adds a function to convert a multicharacter machine to a letter machine (where each transition is exactly one unicode symbol long). It is called fsm_letter_machine() in the API, and _lm() in the built-in functions
+- Print random-lower/random-upper/random-words now provide a "more" random distribution. Duplicated are not printed, but prefixed by a count
+- Thread-safety changes in apply.c API
+- Bug fixes in order of stack operations, printing, reverse operation (.r), function definitions, lower-side priority union (.p.)
+- New API calls fsm_construct_* and fsm_get_* for constructing and reading automata/transducers
+
+0.9.12alpha (20091025)
+
+- OSX version "view" command launches native Graphviz (get it from http://www.pixelglow.com/graphviz/)
+- export cmatrix writes AT&T format weighted transducer
+- More API functions
+- Added support for reading/writing FSM files in AT&T format
+- Added "apply down/up < infile (> outfile)"
+- Separated the API functions into libfoma.h. Separate library builds.
+- Apply functions in API are now iterators
+- Minor bugfixes in regex parsing / lexc file parsing
+- Added the built-in _eq()-operator
+- Added extraction of attested symbol pairs and extraction of sigma as an FSM through "label net" and "sigma net"
+- Minor bugfix in fsm_cross_product()
+- Bugfix in "print dot"
+
+0.9.11alpha (20090722)
+
+- Symbol handling efficiency modifications in fsm_minimize() and fsm_determinize()
+- Added the function fsm_epsilon_remove() (not used outside the API, however)
+- Minor bugfixes in fsm_rewrite()
+- A bug that was reintroduced in fsm_compose() in 0.9.10alpha is fixed
+- fsm_lower() fsm_upper() fsm_kleene_star() fsm_concatenate() have been rewritten
+- Added the functionality "ambiguous upper","extract ambiguous","extract unambiguous" and the corresponding built-in regex functions: _ambdom(), _ambpart(), _unambpart(). The first function extracts the input words that have multiple paths through a transducer. The other two split up a given transducer into an ambiguous or unambiguous one.
+
+0.9.10alpha (20090717)
+- Added support for loading and saving networks in a binary file format and the commands "save stack" "load stack" "save defined" "load defined" and the regular expression construct 'regex @"filename"' which loads the network in "filename" (uses libz)
+- foma is no longer compiled with libgc as default
+- Some major memory management changes in fsm_minimize() and fsm_determinize(). Efficiency/memory tweaks in fsm_determinize().
+- Minor bugfixes in fsm_compose()
+
+0.9.9alpha (20090621)
+- Added support for saving networks in prolog file format (write prolog > filename or wpl > filename).
+- Bugfixes in "read prolog"/"rpl"
+- Minor bugfix in "test unambiguous"/_isunambiguous()
+
+0.9.8alpha (20090604)
+- Added option to load confusion matrices (load cmatrix filename) that specify costs for minimum edit distance matching and attaches to a network. The command "print cmatrix" prints the matrix associated with the network on top of the stack. These are used whenever "apply med" is called. If no matrix is specified, the default distance is Levenshtein.
+- Added the global variables med-cutoff and med-limit to control the med search
+- Minor bugfixes in fsm_compose()
+- Changes in dot file output (print dot, print dot > filename) and "view net"
+- Added tests for transducer ambiguity (test unambiguous/tunam), and the equivalent built-in function _isunambiguous() which returns a boolean automaton.
+
+0.9.7alpha (20090519)
+- Fixed bug in print shortest-string "alias: pss"
+- Added tests for transducer functionality (test functional/tfu) and transducer identity (test identity/tid).
+- Added support for built-in functions. They use the same notation as user-defined ones, except all begin with _. Functions that are network property tests such as _isfunctional() and _isidentity() return boolean automata (the empty set for FALSE and the empty string for TRUE).
+
+0.9.6alpha (20090506)
+- Fixed bugs in apply up
+- Fixed bugs in left and right quotient (\\\ and ///)
+- Changed fsm_minus(), i.e. A - B, so that it subtracts paths instead of being A & ~B. This means transducer paths can be subtracted.
+- Added algorithms for finding the minimum edit distance between a word and a fsm. "apply med" is the same as "apply down" except it finds the cheapest approximate matches. Still experimental.
+
+0.9.5alpha (20090325)
+- Fixed bug in lexcread that affected lexicons with very long words
+- Fixed bug in context restrict with 0 as both contexts, e.g. a => 0 _ 0
+
+0.9.4alpha (20090116)
+- Tweaked memory and efficiency in minimization and determinization algorithms. For large automata, minimization uses much less temporary memory.
+- Added 'flag-is-epsilon' variable behavior and composition algorithm now obeys it
+
+--
+
+0.9.3alpha (20090113)
+- Recursive script-files now work
+- Changed lexc format reading so Lexicon is accepted as well as LEXICON as a keyword
+- Minor bugfix for comment behavior in script files
+- Minor bugfix in "define" format
+
+--
+
+0.9.2alpha (20090111)
+- Determinization now uses much less temporary memory without sacrificing efficiency.
+ The memory use caused severe problems for compiling very large lexicons or determinizing
+ large automata with limited memory.
+
+- Added a "-r" command line option for starting foma without readline (mainly for Win32 version)
+
+- Fixed a serious bug in the ternary `[A,B,C] substitution operator and added the interface command
+ "substitute symbol B for A"
+
+- Added a slight optimization in directed replacements
+
+--
+
+0.9.1alpha (20081231)
+- First public release
diff --git a/COPYING b/COPYING
new file mode 100644
index 0000000..d645695
--- /dev/null
+++ b/COPYING
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/Makefile b/Makefile
new file mode 100644
index 0000000..96e3374
--- /dev/null
+++ b/Makefile
@@ -0,0 +1,110 @@
+prefix = /usr/local
+exec_prefix = $(prefix)
+libdir = $(exec_prefix)/lib
+bindir = $(exec_prefix)/bin
+includedir = $(prefix)/include
+
+VERSION = 0.9.18
+SVN_REV = $(shell svn info | grep "Last Changed Rev" | cut -f4 -d\ )
+CC = gcc
+RANLIB = ranlib
+YACC = bison -v
+LEX = flex -8
+LEXCLEX = flex -8 --prefix=lexc
+LEXIFACE = flex -8 --prefix=interface
+LEXCMATRIX = flex -8 --prefix=cmatrix
+RM = /bin/rm -f
+LDFLAGS = -lreadline -lz -ltermcap
+FLOOKUPLDFLAGS = libfoma.a -lz
+CFLAGS = -O3 -Wall -D_GNU_SOURCE -std=c99 -fvisibility=hidden -fPIC -DSVN_REV=$(SVN_REV)
+FOMAOBJS = foma.o stack.o iface.o lex.interface.o
+LIBOBJS = int_stack.o define.o determinize.o apply.o rewrite.o lexcread.o topsort.o flags.o minimize.o reverse.o extract.o sigma.o io.o structures.o constructions.o coaccessible.o utf8.o spelling.o dynarray.o mem.o stringhash.o trie.o lex.lexc.o lex.yy.o lex.cmatrix.o regex.o
+
+all: libfoma foma flookup cgflookup
+
+foma: $(FOMAOBJS) $(LIBOBJS)
+ $(CC) $(CFLAGS) $(FOMAOBJS) $(LIBOBJS) $(LDFLAGS) -o $@
+
+flookup: flookup.o libfoma
+ $(CC) $(CFLAGS) flookup.o $(FLOOKUPLDFLAGS) -o $@
+
+cgflookup: cgflookup.o libfoma
+ $(CC) $(CFLAGS) cgflookup.o $(FLOOKUPLDFLAGS) -o $@
+
+STATICLIB = libfoma.a
+
+UNAME := $(shell uname)
+
+ifeq ($(UNAME), Darwin)
+ SHAREDLIB = libfoma.dylib
+ SHAREDLIBV = libfoma.$(VERSION).dylib
+ SHAREDLIBM = libfoma.0.dylib
+ AR = libtool
+ ARFLAGS = -static -o
+ DFLAG = -dylib_install_name
+else
+ SHAREDLIB = libfoma.so
+ SHAREDLIBV = libfoma.so.$(VERSION)
+ SHAREDLIBM = libfoma.so.0
+ AR = ar
+ ARFLAGS = cru
+ DFLAG = -soname
+endif
+
+ifeq ($(UNAME), SunOS)
+ DFLAG = -h
+ FLOOKUPLDFLAGS = libfoma.a -lz -lsocket -lnsl
+endif
+
+ifeq ($(UNAME), CYGWIN_NT-5.1)
+ LDFLAGS = /usr/lib/libreadline.dll.a /usr/lib/libz.a
+ FLOOKUPLDFLAGS = libfoma.a /usr/lib/libz.a
+endif
+
+LIBS = $(SHAREDLIBV) $(STATICLIB)
+
+libfoma: $(SHAREDLIBV)
+
+$(SHAREDLIBV): $(LIBOBJS)
+ $(AR) $(ARFLAGS) $(STATICLIB) $(LIBOBJS)
+ $(RANLIB) $(STATICLIB)
+ $(CC) $(CFLAGS) -shared -Wl,$(DFLAG),$(SHAREDLIBM) -o $(SHAREDLIBV) $(LIBOBJS) $(LDFLAGS)
+
+install: foma libfoma
+ - at if [ ! -d $(exec_prefix) ]; then mkdir -p $(exec_prefix); fi
+ - at if [ ! -d $(includedir) ]; then mkdir -p $(includedir); fi
+ - at if [ ! -d $(libdir) ]; then mkdir -p $(libdir); fi
+ - at if [ ! -d $(bindir) ]; then mkdir -p $(bindir); fi
+ cp fomalib.h fomalibconf.h $(includedir);
+ chmod 644 $(includedir)/fomalib.h
+ cp foma flookup cgflookup $(bindir)
+ cp $(LIBS) $(libdir)
+ cd $(libdir); chmod 755 $(LIBS); \
+ if test -f $(libdir)/$(SHAREDLIB); then rm $(libdir)/$(SHAREDLIB); fi
+ if test -f $(libdir)/$(SHAREDLIBM); then rm $(libdir)/$(SHAREDLIBM); fi
+ cd $(libdir); ln -s $(SHAREDLIBV) $(SHAREDLIB); cd $(libdir); \
+ ln -s $(SHAREDLIBV) $(SHAREDLIBM); (ldconfig || true) \
+ >/dev/null 2>&1; \
+
+$(OBJS): foma.h
+
+.c.o:
+ $(CC) $(CFLAGS) -c $< -o $@
+
+lex.yy.c: regex.l regex.h
+ $(LEX) regex.l
+
+lex.lexc.c: lexc.l
+ $(LEXCLEX) $<
+
+lex.interface.c: interface.l
+ $(LEXIFACE) $<
+
+lex.cmatrix.c: cmatrix.l
+ $(LEXCMATRIX) $<
+
+regex.c regex.h: regex.y
+ $(YACC) --defines=$*.h --output=$*.c $<
+
+clean:
+ $(RM) foma flookup cgflookup $(FOMAOBJS) $(LIBOBJS) flookup.o cgflookup.o regex.h regex.c regex.output lex.yy.c lex.lexc.c lex.interface.c lex.cmatrix.c *.so* *.dylib* *.a
diff --git a/README b/README
new file mode 100644
index 0000000..8b15cb2
--- /dev/null
+++ b/README
@@ -0,0 +1,44 @@
+Foma
+====
+
+version 0.9.18alpha
+2015/06/12
+
+Author: Mans Hulden
+Email: mans.hulden at gmail.com
+WWW: http://foma.googlecode.com
+
+
+What is foma?
+-------------
+
+Foma is a multi-purpose finite-state toolkit designed for applications ranging from natural language processing and research in automata theory. It should be upwardly compatible with Xerox xfst and lexc, with the exception of binary file reading and writing.
+
+
+Distribution
+------------
+
+Foma is licensed under the Apache License, version 2. You should have received a copy of the licence with the source code.
+
+
+Compatibility
+-------------
+
+Foma is developed and tested on a Linux system.
+
+It has also been compiled on win32, Mac OSX and Sun Solaris systems. The source code should be reasonably portable. It relies on the GNU readline library. Foma also needs GNU bison (developed using 2.3) and flex (>2.5.31). The last two are only necessary if you are making changes to the parser (.y) or lexer (.l) files.
+
+
+Compiling/installing foma
+-------------------------
+
+A generic Makefile that has been used to compile the Linux, win32, and Mac OSX versions is included. It assumes you have the header files and the necessary libraries mentioned above. Some pre-built binaries are available on http://foma.googlecode.com. Before endeavoring a compile, these are recommended as building for e.g. win32 may be both frustrating and time-consuming.
+
+To compile foma (and flookup) as well as the foma static and dynamic library, "make; make install" should work on most UNIX systems. The default installation target /usr/local can be changed in the Makefile.
+
+
+Bugs
+----
+
+Many. The current release is 0.9.18alpha, and bug reports will be gratefully received at mans.hulden at gmail.com.
+
diff --git a/README.cmatrix b/README.cmatrix
new file mode 100644
index 0000000..d67ea27
--- /dev/null
+++ b/README.cmatrix
@@ -0,0 +1,19 @@
+Since version 0.9.8alpha, foma allows attaching a confusion matrix specification to a network. The command "read cmatrix <filename>" read a confusion matrix and attaches is to the top network on the stack. Subsequent "apply med" commands will use this matrix in determining the minimum cost approximate match to a word. If no confusion matrix is specified, Levenshtein distance is used (insert = substitute = delete = 1).
+
+The command "print cmatrix" also prints out the confusion matrix attached to the top network in tabular format.
+
+The format of the confusion matrix should be clear from the following example confusion matrix file:
+
+--CUT HERE--
+Insert 1
+Substitute 2
+Delete 1
+Cost 1
+a:b c:d
+Cost 3
+:x x: x:y
+--CUT HERE--
+
+The above snippet specifies a matrix where the default insertion cost is 1 unit, the default substitution cost is 2 units, and the default deletion cost is 1 unit. Also, substituting an "a" with a "b" costs 1 unit, as does substituting a "c" for a "d". Inserting an "x" costs 3 units, as does deleting an "x" and substituting an "x" for a "y".
+
+All costs must be positive integer values. A cost specification that involves symbols not found in the alphabet of the top network are not included in the matrix, but are warned about.
diff --git a/README.symbols b/README.symbols
new file mode 100644
index 0000000..c65176c
--- /dev/null
+++ b/README.symbols
@@ -0,0 +1,26 @@
+Foma accepts multiple variants of some operators for compatibility reasons. All symbols in the ASCII range except [1-9A-Za-z'=] are reserved and need to be escaped with % or " " (naturally non-escaped ? is the wildcard symbol and 0 the epsilon symbol).
+
+In addition the following Unicode symbols are reserved and need to be escaped as well, unless used as their operator meaning:
+
+ Octal bytes Character name Hex code point ASCII equivalent
+
+¬ 302 254 NOT SIGN U+00AC ~
+× 303 227 MULTIPLICATION SIGN U+00D7 .x. :
+Σ 316 243 GREEK CAPITAL LETTER SIGMA U+03A3 ?
+ε 316 265 GREEK SMALL LETTER EPSILON U+03B5 [] 0
+→ 342 206 222 RIGHTWARDS ARROW U+2192
+↔ 342 206 224 LEFT RIGHT ARROW U+2194
+∀ 342 210 200 FOR ALL U+2200
+∃ 342 210 203 THERE EXISTS U+2203
+∅ 342 210 205 EMPTY SET U+2205 \?
+∈ 342 210 210 ELEMENT OF U+2208
+∘ 342 210 230 RING OPERATOR U+2218 .o.
+∥ 342 210 245 PARALLEL TO U+2225 <>
+∧ 342 210 247 LOGICAL AND U+2227 &
+∨ 342 210 250 LOGICAL OR U+2228 |
+∩ 342 210 251 INTERSECTION U+2229 &
+∪ 342 210 252 UNION U+222A |
+≤ 342 211 244 LESS-THAN OR EQUAL TO U+2264
+≥ 342 211 245 GREATER-THAN OR EQUAL TO U+2265
+≺ 342 211 272 PRECEDES U+227A <
+≻ 342 211 273 SUCCEEDS U+227B >
diff --git a/apply.c b/apply.c
new file mode 100644
index 0000000..f26c8d4
--- /dev/null
+++ b/apply.c
@@ -0,0 +1,1489 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <time.h>
+#include <string.h>
+#include <limits.h>
+#include "foma.h"
+
+#define RANDOM 1
+#define ENUMERATE 2
+#define MATCH 4
+#define UP 8
+#define DOWN 16
+#define LOWER 32
+#define UPPER 64
+#define SPACE 128
+
+#define FAIL 0
+#define SUCCEED 1
+
+#define DEFAULT_OUTSTRING_SIZE 4096
+#define DEFAULT_STACK_SIZE 128
+
+#define APPLY_BINSEARCH_THRESHOLD 10
+
+#define BITMASK(b) (1 << ((b) & 7))
+#define BITSLOT(b) ((b) >> 3)
+#define BITSET(a,b) ((a)[BITSLOT(b)] |= BITMASK(b))
+#define BITCLEAR(a,b) ((a)[BITSLOT(b)] &= ~BITMASK(b))
+#define BITTEST(a,b) ((a)[BITSLOT(b)] & BITMASK(b))
+#define BITNSLOTS(nb) ((nb + CHAR_BIT - 1) / CHAR_BIT)
+
+
+
+static int apply_append(struct apply_handle *h, int cptr, int sym);
+static char *apply_net(struct apply_handle *h);
+static void apply_create_statemap(struct apply_handle *h,struct fsm *net);
+static void apply_create_sigarray(struct apply_handle *h,struct fsm *net);
+static void apply_create_sigmatch(struct apply_handle *h);
+int apply_match_length(struct apply_handle *h, int symbol);
+static int apply_match_str(struct apply_handle *h,int symbol, int position);
+static void apply_add_flag(struct apply_handle *h,char *name);
+static int apply_check_flag(struct apply_handle *h,int type, char *name, char *value);
+static void apply_clear_flags(struct apply_handle *h);
+void apply_set_iptr(struct apply_handle *h);
+void apply_mark_flagstates(struct apply_handle *h);
+void apply_clear_index(struct apply_handle *h);
+
+static void apply_stack_clear(struct apply_handle *h);
+static int apply_stack_isempty(struct apply_handle *h);
+static void apply_stack_pop (struct apply_handle *h);
+static void apply_stack_push (struct apply_handle *h, int vmark, char *sflagname, char *sflagvalue, int sflagneg);
+static void apply_force_clear_stack(struct apply_handle *h);
+
+
+void apply_set_obey_flags(struct apply_handle *h, int value) {
+ h->obey_flags = value;
+}
+
+void apply_set_show_flags(struct apply_handle *h, int value) {
+ h->show_flags = value;
+}
+
+void apply_set_print_space(struct apply_handle *h, int value) {
+ h->print_space = value;
+ h->space_symbol = strdup(" ");
+}
+
+void apply_set_separator(struct apply_handle *h, char *symbol) {
+ h->separator = strdup(symbol);
+}
+
+void apply_set_epsilon(struct apply_handle *h, char *symbol) {
+ xxfree(h->epsilon_symbol);
+ h->epsilon_symbol = strdup(symbol);
+ (h->sigs+EPSILON)->symbol = h->epsilon_symbol;
+ (h->sigs+EPSILON)->length = strlen(h->epsilon_symbol);
+}
+
+void apply_set_space_symbol(struct apply_handle *h, char *space) {
+ h->space_symbol = strdup(space);
+ h->print_space = 1;
+}
+
+void apply_set_print_pairs(struct apply_handle *h, int value) {
+ h->print_pairs = value;
+}
+
+static void apply_force_clear_stack(struct apply_handle *h) {
+ /* Make sure stack is empty and marks reset */
+ if (!apply_stack_isempty(h)) {
+ *(h->marks+(h->gstates+h->ptr)->state_no) = 0;
+ while (!apply_stack_isempty(h)) {
+ apply_stack_pop(h);
+ *(h->marks+(h->gstates+h->ptr)->state_no) = 0;
+ }
+ h->iterator = 0;
+ h->iterate_old = 0;
+ apply_stack_clear(h);
+ }
+}
+
+char *apply_enumerate(struct apply_handle *h) {
+
+ char *result = NULL;
+
+ if (h->last_net == NULL || h->last_net->finalcount == 0) {
+ return (NULL);
+ }
+ h->binsearch = 0;
+ if (h->iterator == 0) {
+ h->iterate_old = 0;
+ apply_force_clear_stack(h);
+ result = apply_net(h);
+ if ((h->mode & RANDOM) != RANDOM)
+ (h->iterator)++;
+ } else {
+ h->iterate_old = 1;
+ result = apply_net(h);
+ }
+ return(result);
+}
+
+char *apply_words(struct apply_handle *h) {
+ h->mode = DOWN + ENUMERATE + LOWER + UPPER;
+ return(apply_enumerate(h));
+}
+
+char *apply_upper_words(struct apply_handle *h) {
+ h->mode = DOWN + ENUMERATE + UPPER;
+ return(apply_enumerate(h));
+}
+
+char *apply_lower_words(struct apply_handle *h) {
+ h->mode = DOWN + ENUMERATE + LOWER;
+ return(apply_enumerate(h));
+}
+
+char *apply_random_words(struct apply_handle *h) {
+ apply_clear_flags(h);
+ h->mode = DOWN + ENUMERATE + LOWER + UPPER + RANDOM;
+ return(apply_enumerate(h));
+}
+
+char *apply_random_lower(struct apply_handle *h) {
+ apply_clear_flags(h);
+ h->mode = DOWN + ENUMERATE + LOWER + RANDOM;
+ return(apply_enumerate(h));
+}
+
+char *apply_random_upper(struct apply_handle *h) {
+ apply_clear_flags(h);
+ h->mode = DOWN + ENUMERATE + UPPER + RANDOM;
+ return(apply_enumerate(h));
+}
+
+/* Frees memory associated with applies */
+void apply_clear(struct apply_handle *h) {
+ struct sigma_trie_arrays *sta, *stap;
+ for (sta = h->sigma_trie_arrays; sta != NULL; ) {
+ stap = sta;
+ xxfree(sta->arr);
+ sta = sta->next;
+ xxfree(stap);
+ }
+ h->sigma_trie_arrays = NULL;
+ if (h->statemap != NULL) {
+ xxfree(h->statemap);
+ h->statemap = NULL;
+ }
+ if (h->numlines != NULL) {
+ xxfree(h->numlines);
+ h->numlines = NULL;
+ }
+ if (h->marks != NULL) {
+ xxfree(h->marks);
+ h->marks = NULL;
+ }
+ if (h->searchstack != NULL) {
+ xxfree(h->searchstack);
+ h->searchstack = NULL;
+ }
+ if (h->sigs != NULL) {
+ xxfree(h->sigs);
+ h->sigs = NULL;
+ }
+ if (h->flag_lookup != NULL) {
+ xxfree(h->flag_lookup);
+ h->flag_lookup = NULL;
+ }
+ if (h->sigmatch_array != NULL) {
+ xxfree(h->sigmatch_array);
+ h->sigmatch_array = NULL;
+ }
+ if (h->flagstates != NULL) {
+ xxfree(h->flagstates);
+ h->flagstates = NULL;
+ }
+ apply_clear_index(h);
+ h->last_net = NULL;
+ h->iterator = 0;
+ xxfree(h->outstring);
+ xxfree(h->separator);
+ xxfree(h->epsilon_symbol);
+ xxfree(h);
+}
+
+char *apply_updown(struct apply_handle *h, char *word) {
+
+ char *result = NULL;
+
+ if (h->last_net == NULL || h->last_net->finalcount == 0)
+ return (NULL);
+
+ if (word == NULL) {
+ h->iterate_old = 1;
+ result = apply_net(h);
+ }
+ else if (word != NULL) {
+ h->iterate_old = 0;
+ h->instring = word;
+ apply_create_sigmatch(h);
+
+ /* Remove old marks if necessary TODO: only pop marks */
+ apply_force_clear_stack(h);
+ result = apply_net(h);
+ }
+ return(result);
+}
+
+char *apply_down(struct apply_handle *h, char *word) {
+
+ h->mode = DOWN;
+ if (h->index_in) {
+ h->indexed = 1;
+ } else {
+ h->indexed = 0;
+ }
+ h->binsearch = (h->last_net->arcs_sorted_in == 1) ? 1 : 0;
+ return(apply_updown(h, word));
+}
+
+char *apply_up(struct apply_handle *h, char *word) {
+
+ h->mode = UP;
+ if (h->index_out) {
+ h->indexed = 1;
+ } else {
+ h->indexed = 0;
+ }
+ h->binsearch = (h->last_net->arcs_sorted_out == 1) ? 1 : 0;
+ return(apply_updown(h, word));
+}
+
+struct apply_handle *apply_init(struct fsm *net) {
+ struct apply_handle *h;
+
+ srand((unsigned int) time(NULL));
+ h = calloc(1,sizeof(struct apply_handle));
+ /* Init */
+
+ h->iterate_old = 0;
+ h->iterator = 0;
+ h->instring = NULL;
+ h->flag_list = NULL;
+ h->flag_lookup = NULL;
+ h->obey_flags = 1;
+ h->show_flags = 0;
+ h->print_space = 0;
+ h->print_pairs = 0;
+ h->separator = strdup(":");
+ h->epsilon_symbol = strdup("0");
+ h->last_net = net;
+ h->outstring = xxmalloc(sizeof(char)*DEFAULT_OUTSTRING_SIZE);
+ h->outstringtop = DEFAULT_OUTSTRING_SIZE;
+ *(h->outstring) = '\0';
+ h->gstates = net->states;
+ h->gsigma = net->sigma;
+ h->printcount = 1;
+ apply_create_statemap(h, net);
+ h->searchstack = xxmalloc(sizeof(struct searchstack) * DEFAULT_STACK_SIZE);
+ h->apply_stack_top = DEFAULT_STACK_SIZE;
+ apply_stack_clear(h);
+ apply_create_sigarray(h, net);
+ return(h);
+}
+
+int apply_stack_isempty (struct apply_handle *h) {
+ if (h->apply_stack_ptr == 0) {
+ return 1;
+ }
+ return 0;
+}
+
+void apply_stack_clear (struct apply_handle *h) {
+ h->apply_stack_ptr = 0;
+}
+
+void apply_stack_pop (struct apply_handle *h) {
+ struct flag_list *flist;
+ struct searchstack *ss;
+ (h->apply_stack_ptr)--;
+ ss = h->searchstack+h->apply_stack_ptr;
+
+ h->iptr = ss->iptr;
+ h->ptr = ss->offset;
+ h->ipos = ss->ipos;
+ h->opos = ss->opos;
+ h->state_has_index = ss->state_has_index;
+ /* Restore mark */
+ *(h->marks+(h->gstates+h->ptr)->state_no) = ss->visitmark;
+
+ if (h->has_flags && ss->flagname != NULL) {
+ /* Restore flag */
+ for (flist = h->flag_list; flist != NULL; flist = flist->next) {
+ if (strcmp(flist->name, ss->flagname) == 0) {
+ break;
+ }
+ }
+ if (flist == NULL)
+ perror("***Nothing to pop\n");
+ flist->value = ss->flagvalue;
+ flist->neg = ss->flagneg;
+ }
+}
+
+static void apply_stack_push (struct apply_handle *h, int vmark, char *sflagname, char *sflagvalue, int sflagneg) {
+ struct searchstack *ss;
+ if (h->apply_stack_ptr == h->apply_stack_top) {
+ h->searchstack = xxrealloc(h->searchstack, sizeof(struct searchstack)* ((h->apply_stack_top)*2));
+ if (h->searchstack == NULL) {
+ perror("Apply stack full!!!\n");
+ exit(0);
+ }
+ h->apply_stack_top *= 2;
+ }
+ ss = h->searchstack+h->apply_stack_ptr;
+ ss->offset = h->curr_ptr;
+ ss->ipos = h->ipos;
+ ss->opos = h->opos;
+ ss->visitmark = vmark;
+ ss->iptr = h->iptr;
+ ss->state_has_index = h->state_has_index;
+ if (h->has_flags) {
+ ss->flagname = sflagname;
+ ss->flagvalue = sflagvalue;
+ ss->flagneg = sflagneg;
+ }
+ (h->apply_stack_ptr)++;
+}
+
+void apply_reset_enumerator(struct apply_handle *h) {
+ int statecount, i;
+ statecount = h->last_net->statecount;
+ for (i=0; i < statecount; i++) {
+ *(h->marks+i) = 0;
+ }
+ h->iterator = 0;
+ h->iterate_old = 0;
+}
+
+void apply_clear_index_list(struct apply_handle *h, struct apply_state_index **index) {
+ int i, j, statecount;
+ struct apply_state_index *iptr, *iptr_tmp, *iptr_zero;
+ if (index == NULL)
+ return;
+ statecount = h->last_net->statecount;
+ for (i = 0; i < statecount; i++) {
+ iptr = *(index+i);
+ if (iptr == NULL) {
+ continue;
+ }
+ iptr_zero = *(index+i);
+ for (j = h->sigma_size - 1 ; j >= 0; j--) { /* Make sure to not free the list in EPSILON */
+ iptr = *(index+i) + j; /* as the other states lists' tails point to it */
+ for (iptr = iptr->next ; iptr != NULL && iptr != iptr_zero; iptr = iptr_tmp) {
+ iptr_tmp = iptr->next;
+ xxfree(iptr);
+ }
+ }
+ xxfree(*(index+i));
+ }
+}
+
+void apply_clear_index(struct apply_handle *h) {
+ if (h->index_in) {
+ apply_clear_index_list(h, h->index_in);
+ xxfree(h->index_in);
+ h->index_in = NULL;
+ }
+ if (h->index_out) {
+ apply_clear_index_list(h, h->index_out);
+ xxfree(h->index_out);
+ h->index_out = NULL;
+ }
+}
+
+void apply_index(struct apply_handle *h, int inout, int densitycutoff, int mem_limit, int flags_only) {
+ struct fsm_state *fsm;
+ unsigned int cnt = 0;
+ int i, j, maxtrans, numtrans, laststate, sym;
+ fsm = h->gstates;
+
+ struct apply_state_index **indexptr, *iptr, *tempiptr;
+
+ struct pre_index {
+ int state_no;
+ struct pre_index *next;
+ } *pre_index, *tp, *tpp;
+ if (flags_only && !h->has_flags) {
+ return;
+ }
+ /* get numtrans */
+ for (i=0, laststate = 0, maxtrans = 0, numtrans = 0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->state_no != laststate) {
+ maxtrans = numtrans > maxtrans ? numtrans : maxtrans;
+ numtrans = 0;
+ }
+ if ((fsm+i)->target != -1) {
+ numtrans++;
+ }
+ laststate = (fsm+i)->state_no;
+ }
+
+ pre_index = xxcalloc(maxtrans+1, sizeof(struct pre_index));
+ for (i = 0; i <= maxtrans; i++) {
+ (pre_index+i)->state_no = -1;
+ }
+
+ /* We create an array of states, indexed by how many transitions they have */
+ /* so that later, we can traverse them in order densest first, in case we */
+ /* only want to index to some predefined maximum memory usage. */
+
+ for (i = 0, laststate = 0, maxtrans = 0, numtrans = 0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->state_no != laststate) {
+ if ((pre_index+numtrans)->state_no == -1) {
+ (pre_index+numtrans)->state_no = laststate;
+ } else {
+ tp = xxcalloc(1, sizeof(struct pre_index));
+ tp->state_no = laststate;
+ tp->next = (pre_index+numtrans)->next;
+ (pre_index+numtrans)->next = tp;
+ }
+ maxtrans = numtrans > maxtrans ? numtrans : maxtrans;
+ numtrans = 0;
+ }
+ if ((fsm+i)->target != -1) {
+ numtrans++;
+ }
+ laststate = (fsm+i)->state_no;
+ }
+ indexptr = NULL;
+ cnt += round_up_to_power_of_two(h->last_net->statecount*sizeof(struct apply_state_index *));
+
+ if (cnt > mem_limit) {
+ cnt -= round_up_to_power_of_two(h->last_net->statecount*sizeof(struct apply_state_index *));
+ goto memlimitnoindex;
+ }
+
+ indexptr = xxcalloc(h->last_net->statecount, sizeof(struct apply_state_index *));
+
+ if (h->has_flags && flags_only) {
+ /* Mark states that have flags */
+ if (!(h->flagstates)) {
+ apply_mark_flagstates(h);
+ }
+ }
+
+ for (i = maxtrans; i >= 0; i--) {
+ for (tp = pre_index+i; tp != NULL; tp = tp->next) {
+ if (tp->state_no >= 0) {
+ if (i < densitycutoff) {
+ if (!(h->has_flags && flags_only && BITTEST(h->flagstates, tp->state_no))) {
+ continue;
+ }
+ }
+ cnt += round_up_to_power_of_two(h->sigma_size*sizeof(struct apply_state_index));
+ if (cnt > mem_limit) {
+ cnt -= round_up_to_power_of_two(h->sigma_size*sizeof(struct apply_state_index));
+ goto memlimit;
+ }
+ *(indexptr + tp->state_no) = xxmalloc(h->sigma_size*sizeof(struct apply_state_index));
+
+ /* We make the tail of all index linked lists point to the index */
+ /* for EPSILON, so that we automatically when EPSILON transitions */
+ /* also when traversing an index. */
+
+ for (j = 0; j < h->sigma_size; j++) {
+ (*(indexptr + tp->state_no) + j)->fsmptr = -1;
+ if (j == EPSILON)
+ (*(indexptr + tp->state_no) + j)->next = NULL;
+ else
+ (*(indexptr + tp->state_no) + j)->next = (*(indexptr + tp->state_no)); /* all tails point to epsilon */
+ }
+ }
+ }
+ }
+
+ memlimit:
+
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ iptr = *(indexptr + (fsm+i)->state_no);
+ if (iptr == NULL || (fsm+i)->target == -1) {
+ continue;
+ }
+ sym = inout == APPLY_INDEX_INPUT ? (fsm+i)->in : (fsm+i)->out;
+
+ if (h->has_flags && (h->flag_lookup+sym)->type) {
+ sym = EPSILON;
+ }
+ if (sym == UNKNOWN) { /* We make the index of UNKNOWN point to IDENTITY */
+ sym = IDENTITY; /* since these are really the same symbol */
+ }
+ if ((iptr+sym)->fsmptr == -1) {
+ (iptr+sym)->fsmptr = i;
+ } else {
+ cnt += round_up_to_power_of_two(sizeof(struct apply_state_index));
+ tempiptr = xxcalloc(1, sizeof(struct apply_state_index));
+
+ tempiptr->next = (iptr+sym)->next;
+ tempiptr->fsmptr = i;
+ (iptr+sym)->next = tempiptr;
+ }
+ }
+
+ /* Free preindex */
+
+ memlimitnoindex:
+
+ for (i = maxtrans; i >= 0; i--) {
+ for (tp = (pre_index+i)->next; tp != NULL; tp = tpp) {
+ tpp = tp->next;
+ xxfree(tp);
+ }
+ }
+ xxfree(pre_index);
+
+ if (inout == APPLY_INDEX_INPUT) {
+ h->index_in = indexptr;
+ } else {
+ h->index_out = indexptr;
+ }
+}
+
+int apply_binarysearch(struct apply_handle *h) {
+ int thisstate, nextsym, seeksym, thisptr, lastptr, midptr;
+
+ thisptr = h->curr_ptr = h->ptr;
+ nextsym = (((h->mode) & DOWN) == DOWN) ? (h->gstates+h->curr_ptr)->in : (h->gstates+h->curr_ptr)->out;
+ if (nextsym == EPSILON)
+ return 1;
+ if (nextsym == -1)
+ return 0;
+ if (h->ipos >= h->current_instring_length) {
+ return 0;
+ }
+ seeksym = (h->sigmatch_array+h->ipos)->signumber;
+ if (seeksym == nextsym || (nextsym == UNKNOWN && seeksym == IDENTITY))
+ return 1;
+
+ thisstate = (h->gstates+thisptr)->state_no;
+ lastptr = *(h->statemap+thisstate)+*(h->numlines+thisstate)-1;
+ thisptr++;
+
+ if (seeksym == IDENTITY || lastptr - thisptr < APPLY_BINSEARCH_THRESHOLD) {
+ for ( ; thisptr <= lastptr; thisptr++) {
+ nextsym = (((h->mode) & DOWN) == DOWN) ? (h->gstates+thisptr)->in : (h->gstates+thisptr)->out;
+ if ((nextsym == seeksym) || (nextsym == UNKNOWN && seeksym == IDENTITY)) {
+ h->curr_ptr = thisptr;
+ return 1;
+ }
+ if (nextsym > seeksym || nextsym == -1) {
+ return 0;
+ }
+ }
+ return 0;
+ }
+
+ for (;;) {
+ if (thisptr > lastptr) { return 0; }
+ midptr = (thisptr+lastptr)/2;
+ nextsym = (((h->mode) & DOWN) == DOWN) ? (h->gstates+midptr)->in : (h->gstates+midptr)->out;
+ if (seeksym < nextsym) {
+ lastptr = midptr - 1;
+ continue;
+ } else if (seeksym > nextsym) {
+ thisptr = midptr + 1;
+ continue;
+ } else {
+
+ while (((((h->mode) & DOWN) == DOWN) ? (h->gstates+(midptr-1))->in : (h->gstates+(midptr-1))->out) == seeksym) {
+ midptr--; /* Find first match in case of ties */
+ }
+ h->curr_ptr = midptr;
+ return 1;
+ }
+ }
+}
+
+int apply_follow_next_arc(struct apply_handle *h) {
+ char *fname, *fvalue;
+ int eatupi, eatupo, symin, symout, fneg;
+ int vcount, marksource, marktarget;
+
+ /* Here we follow three possible search strategies: */
+ /* (1) if the state in question has an index, we use that */
+ /* (2) if the state is binary searchable, we use that */
+ /* (3) otherwise we traverse arc-by-arc */
+ /* Condition (2) needs arcs to be sorted in the proper */
+ /* direction, and requires that the state be flag-free */
+ /* For those states that aren't flag-free, (3) is used */
+
+ if (h->state_has_index) {
+ for ( ; h->iptr != NULL && h->iptr->fsmptr != -1; ) {
+
+ h->ptr = h->curr_ptr = h->iptr->fsmptr;
+ if (((h->mode) & DOWN) == DOWN) {
+ symin = (h->gstates+h->curr_ptr)->in;
+ symout = (h->gstates+h->curr_ptr)->out;
+ } else {
+ symin = (h->gstates+h->curr_ptr)->out;
+ symout = (h->gstates+h->curr_ptr)->in;
+ }
+
+ marksource = *(h->marks+(h->gstates+h->ptr)->state_no);
+ marktarget = *(h->marks+(h->gstates+(*(h->statemap+(h->gstates+h->curr_ptr)->target)))->state_no);
+ eatupi = apply_match_length(h, symin);
+ if (!(eatupi == -1 || -1-(h->ipos)-eatupi == marktarget)) { /* input 2x EPSILON loop check */
+ if ((eatupi = apply_match_str(h, symin, h->ipos)) != -1) {
+ eatupo = apply_append(h, h->curr_ptr, symout);
+ if (h->obey_flags && h->has_flags && ((h->flag_lookup+symin)->type & (FLAG_UNIFY|FLAG_CLEAR|FLAG_POSITIVE|FLAG_NEGATIVE))) {
+ fname = (h->flag_lookup+symin)->name;
+ fvalue = h->oldflagvalue;
+ fneg = h->oldflagneg;
+ } else {
+ fname = fvalue = NULL;
+ fneg = 0;
+ }
+ /* Push old position */
+ apply_stack_push(h, marksource, fname, fvalue, fneg);
+ h->ptr = *(h->statemap+(h->gstates+h->curr_ptr)->target);
+ h->ipos += eatupi;
+ h->opos += eatupo;
+ apply_set_iptr(h);
+ return 1;
+ }
+ }
+ h->iptr = h->iptr->next;
+ }
+ return 0;
+ } else if ((h->binsearch && !(h->has_flags)) || (h->binsearch && !(BITTEST(h->flagstates, (h->gstates+h->ptr)->state_no)))) {
+ for (;;) {
+ if (apply_binarysearch(h)) {
+ if (((h->mode) & DOWN) == DOWN) {
+ symin = (h->gstates+h->curr_ptr)->in;
+ symout = (h->gstates+h->curr_ptr)->out;
+ } else {
+ symin = (h->gstates+h->curr_ptr)->out;
+ symout = (h->gstates+h->curr_ptr)->in;
+ }
+
+ marksource = *(h->marks+(h->gstates+h->ptr)->state_no);
+ marktarget = *(h->marks+(h->gstates+(*(h->statemap+(h->gstates+h->curr_ptr)->target)))->state_no);
+
+ eatupi = apply_match_length(h, symin);
+ if (eatupi != -1 && -1-(h->ipos)-eatupi != marktarget) {
+ if ((eatupi = apply_match_str(h, symin, h->ipos)) != -1) {
+ eatupo = apply_append(h, h->curr_ptr, symout);
+
+ /* Push old position */
+ apply_stack_push(h, marksource, NULL, NULL, 0);
+
+ /* Follow arc */
+ h->ptr = *(h->statemap+(h->gstates+h->curr_ptr)->target);
+ h->ipos += eatupi;
+ h->opos += eatupo;
+ apply_set_iptr(h);
+ return 1;
+ }
+ }
+ if ((h->gstates+h->curr_ptr)->state_no == (h->gstates+h->curr_ptr+1)->state_no) {
+ h->curr_ptr++;
+ h->ptr = h->curr_ptr;
+ if ((h->gstates+h->curr_ptr)-> target == -1) {
+ return 0;
+ }
+ continue;
+ }
+ }
+ return 0;
+ }
+ } else {
+ for (h->curr_ptr = h->ptr; (h->gstates+h->curr_ptr)->state_no == (h->gstates+h->ptr)->state_no && (h->gstates+h->curr_ptr)-> in != -1; (h->curr_ptr)++) {
+
+ /* Select one random arc to follow out of all outgoing arcs */
+ if ((h->mode & RANDOM) == RANDOM) {
+ vcount = 0;
+ for (h->curr_ptr = h->ptr; (h->gstates+h->curr_ptr)->state_no == (h->gstates+h->ptr)->state_no && (h->gstates+h->curr_ptr)-> in != -1; (h->curr_ptr)++) {
+ vcount++;
+ }
+ if (vcount > 0) {
+ h->curr_ptr = h->ptr + (rand() % vcount);
+ } else {
+ h->curr_ptr = h->ptr;
+ }
+ }
+
+ if (((h->mode) & DOWN) == DOWN) {
+ symin = (h->gstates+h->curr_ptr)->in;
+ symout = (h->gstates+h->curr_ptr)->out;
+ } else {
+ symin = (h->gstates+h->curr_ptr)->out;
+ symout = (h->gstates+h->curr_ptr)->in;
+ }
+
+ marksource = *(h->marks+(h->gstates+h->ptr)->state_no);
+ marktarget = *(h->marks+(h->gstates+(*(h->statemap+(h->gstates+h->curr_ptr)->target)))->state_no);
+
+ eatupi = apply_match_length(h, symin);
+
+ if (eatupi == -1 || -1-(h->ipos)-eatupi == marktarget) { continue; } /* loop check */
+ if ((eatupi = apply_match_str(h, symin, h->ipos)) != -1) {
+ eatupo = apply_append(h, h->curr_ptr, symout);
+ if (h->obey_flags && h->has_flags && ((h->flag_lookup+symin)->type & (FLAG_UNIFY|FLAG_CLEAR|FLAG_POSITIVE|FLAG_NEGATIVE))) {
+
+ fname = (h->flag_lookup+symin)->name;
+ fvalue = h->oldflagvalue;
+ fneg = h->oldflagneg;
+ } else {
+ fname = fvalue = NULL;
+ fneg = 0;
+ }
+
+ /* Push old position */
+ apply_stack_push(h, marksource, fname, fvalue, fneg);
+
+ /* Follow arc */
+ h->ptr = *(h->statemap+(h->gstates+h->curr_ptr)->target);
+ h->ipos += eatupi;
+ h->opos += eatupo;
+ apply_set_iptr(h);
+ return(1);
+ }
+ }
+ return(0);
+ }
+}
+
+char *apply_return_string(struct apply_handle *h) {
+ /* Stick a 0 to endpos to avoid getting old accumulated gunk strings printed */
+ *(h->outstring+h->opos) = '\0';
+ if (((h->mode) & RANDOM) == RANDOM) {
+ /* To end or not to end */
+ if (!(rand() % 2)) {
+ apply_stack_clear(h);
+ h->iterator = 0;
+ h->iterate_old = 0;
+ return(h->outstring);
+ }
+ } else {
+ return(h->outstring);
+ }
+ return(NULL);
+}
+
+void apply_mark_state(struct apply_handle *h) {
+
+ /* This controls the how epsilon-loops are traversed. Such loops can */
+ /* only be followed once to reach a state already visited in the DFS. */
+ /* This requires that we store the number of input symbols consumed */
+ /* whenever we enter a new state. If we enter the same state twice */
+ /* with the same number of input symbols consumed, we abandon the search */
+ /* for that branch. Flags are epsilons from this point of view. */
+ /* The encoding of h->marks is: */
+ /* 0 = unseen, +ipos = seen at ipos, -ipos = seen second time at ipos */
+
+ if ((h->mode & RANDOM) != RANDOM) {
+ if (*(h->marks+(h->gstates+h->ptr)->state_no) == h->ipos+1) {
+ *(h->marks+(h->gstates+h->ptr)->state_no) = -(h->ipos+1);
+ } else {
+ *(h->marks+(h->gstates+h->ptr)->state_no) = h->ipos+1;
+ }
+ }
+}
+
+void apply_skip_this_arc(struct apply_handle *h) {
+ /* If we have index ptr */
+ if (h->iptr) {
+ h->ptr = h->iptr->fsmptr;
+ h->iptr = h->iptr->next;
+ /* Otherwise */
+ } else {
+ (h->ptr)++;
+ }
+}
+
+int apply_at_last_arc(struct apply_handle *h) {
+ int seeksym, nextsym;
+ if (h->state_has_index) {
+ if (h->iptr->next == NULL || h->iptr->next->fsmptr == -1) {
+ return 1;
+ }
+ } else {
+ if ((h->binsearch && !(h->has_flags)) || (h->binsearch && !(BITTEST(h->flagstates, (h->gstates+h->ptr)->state_no)))) {
+ if ((h->gstates+h->ptr)->state_no != (h->gstates+h->ptr+1)->state_no) {
+ return 1;
+ }
+ seeksym = (h->sigmatch_array+h->ipos)->signumber;
+ nextsym = (((h->mode) & DOWN) == DOWN) ? (h->gstates+h->ptr)->in : (h->gstates+h->ptr)->out;
+ if (nextsym == -1 || seeksym < nextsym) {
+ return 1;
+ }
+ } else {
+ if ((h->gstates+h->ptr)->state_no != (h->gstates+h->ptr+1)->state_no) {
+ return 1;
+ }
+ }
+ }
+ return 0;
+}
+
+/* map h->ptr (line pointer) to h->iptr (index pointer) */
+void apply_set_iptr(struct apply_handle *h) {
+ struct apply_state_index **idx, *sidx;
+ int stateno, seeksym;
+ /* Check if state has index */
+ if ((idx = ((h->mode) & DOWN) == DOWN ? (h->index_in) : (h->index_out)) == NULL) {
+ return;
+ }
+
+ h->iptr = NULL;
+ h->state_has_index = 0;
+ stateno = (h->gstates+h->ptr)->state_no;
+ if (stateno < 0) {
+ return;
+ }
+
+ sidx = *(idx + stateno);
+ if (sidx == NULL) { return; }
+ seeksym = (h->sigmatch_array+h->ipos)->signumber;
+ h->state_has_index = 1;
+ sidx = sidx + seeksym;
+ if (sidx->fsmptr == -1) {
+ if (sidx->next == NULL) {
+ return;
+ } else {
+ sidx = sidx->next;
+ }
+ }
+ h->iptr = sidx;
+ if (sidx->fsmptr == -1) {
+ h->iptr = NULL;
+ }
+ h->state_has_index = 1;
+}
+
+char *apply_net(struct apply_handle *h) {
+
+/* We perform a basic DFS on the graph, with two minor complications: */
+
+/* 1. We keep a mark for each state which indicates how many input symbols */
+/* we had consumed the last time we entered that state on the current */
+/* "run." If we reach a state seen twice without consuming input, we */
+/* terminate that branch of the search. */
+/* As we pop a position, we also unmark the state we came from. */
+
+/* 2. If the graph has flags, we push the previous flag value when */
+/* traversing a flag-modifying arc (P,U,N, or C). This is because a */
+/* flag may have been set during the previous "run" and may not apply. */
+/* Since we're doing a DFS, we can be sure to return to the previous */
+/* global flag state by just remembering that last flag change. */
+
+/* 3. The whole system needs to work as an iterator, meaning we need to */
+/* store the global state of the search so we can resume it later to */
+/* to yield more possible output words with the same input string. */
+
+ char *returnstring;
+
+ if (h->iterate_old == 1) { /* If called with NULL as the input word, this will be set */
+ goto resume;
+ }
+
+ h->iptr = NULL; h->ptr = 0; h->ipos = 0; h->opos = 0;
+ apply_set_iptr(h);
+
+ apply_stack_clear(h);
+
+ if (h->has_flags) {
+ apply_clear_flags(h);
+ }
+
+ /* "The use of four-letter words like goto can occasionally be justified */
+ /* even in the best of company." Knuth (1974). */
+
+ goto L2;
+
+ while(!apply_stack_isempty(h)) {
+ apply_stack_pop(h);
+ /* If last line was popped */
+ if (apply_at_last_arc(h)) {
+ *(h->marks+(h->gstates+h->ptr)->state_no) = 0; /* Unmark */
+ continue; /* pop next */
+ }
+ apply_skip_this_arc(h); /* skip old pushed arc */
+ L1:
+ if (!apply_follow_next_arc(h)) {
+ *(h->marks+(h->gstates+h->ptr)->state_no) = 0; /* Unmark */
+ continue; /* pop next */
+ }
+ L2:
+ /* Print accumulated string upon entry to state */
+ if ((h->gstates+h->ptr)->final_state == 1 && (h->ipos == h->current_instring_length || ((h->mode) & ENUMERATE) == ENUMERATE)) {
+ if ((returnstring = (apply_return_string(h))) != NULL) {
+ return(returnstring);
+ }
+ }
+
+ resume:
+ apply_mark_state(h); /* Mark upon arrival to new state */
+ goto L1;
+ }
+ if ((h->mode & RANDOM) == RANDOM) {
+ apply_stack_clear(h);
+ h->iterator = 0;
+ h->iterate_old = 0;
+ return(h->outstring);
+ }
+ apply_stack_clear(h);
+ return NULL;
+}
+
+int apply_append(struct apply_handle *h, int cptr, int sym) {
+
+ char *astring, *bstring, *pstring;
+ int symin, symout, len, alen, blen, idlen;
+
+ symin = (h->gstates+cptr)->in;
+ symout = (h->gstates+cptr)->out;
+ astring = ((h->sigs)+symin)->symbol;
+ alen = ((h->sigs)+symin)->length;
+ bstring = ((h->sigs)+symout)->symbol;
+ blen = ((h->sigs)+symout)->length;
+
+ while (alen + blen + h->opos + 2 + strlen(h->separator) >= h->outstringtop) {
+ // while (alen + blen + h->opos + 3 >= h->outstringtop) {
+ h->outstring = xxrealloc(h->outstring, sizeof(char) * ((h->outstringtop) * 2));
+ (h->outstringtop) *= 2;
+ }
+
+ if ((h->has_flags) && !h->show_flags && (h->flag_lookup+symin)->type) {
+ astring = ""; alen = 0;
+ }
+ if (h->has_flags && !h->show_flags && (h->flag_lookup+symout)->type) {
+ bstring = ""; blen = 0;
+ }
+ if (((h->mode) & ENUMERATE) == ENUMERATE) {
+ /* Print both sides separated by colon */
+ /* if we're printing "words" */
+ if (((h->mode) & (UPPER | LOWER)) == (UPPER|LOWER)) {
+
+ if (astring == bstring) {
+ strcpy(h->outstring+h->opos, astring);
+ len = alen;
+ } else {
+ strcpy(h->outstring+h->opos, astring);
+ // strcpy(h->outstring+h->opos+alen,":");
+ strcpy(h->outstring+h->opos+alen,h->separator);
+ //strcpy(h->outstring+h->opos+alen+1,bstring);
+ strcpy(h->outstring+h->opos+alen+strlen(h->separator),bstring);
+ // len = alen+blen+1;
+ len = alen+blen+strlen(h->separator);
+ }
+ }
+
+ /* Print one side only */
+ if (((h->mode) & (UPPER|LOWER)) != (UPPER|LOWER)) {
+
+ if (symin == EPSILON) {
+ astring = ""; alen = 0;
+ }
+ if (symout == EPSILON) {
+ bstring = ""; blen = 0;
+ }
+ if (((h->mode) & (UPPER|LOWER)) == UPPER) {
+ pstring = astring;
+ len = alen;
+ } else {
+ pstring = bstring;
+ len = blen;
+ }
+ //strcpy(h->outstring+h->opos, pstring);
+ memcpy(h->outstring+h->opos, pstring, len);
+ }
+ }
+ if (((h->mode) & ENUMERATE) != ENUMERATE) {
+ /* Print pairs is ON and symbols are different */
+ if (h->print_pairs && (symin != symout)) {
+
+ if (symin == UNKNOWN && ((h->mode) & DOWN) == DOWN)
+ strncpy(astring, h->instring+h->ipos, 1);
+ if (symout == UNKNOWN && ((h->mode) & UP) == UP)
+ strncpy(bstring, h->instring+h->ipos, 1);
+ strcpy(h->outstring+h->opos, "<");
+ strcpy(h->outstring+h->opos+1, astring);
+ //strcpy(h->outstring+h->opos+alen+1,":");
+ strcpy(h->outstring+h->opos+alen+1,h->separator);
+ //strcpy(h->outstring+h->opos+alen+2,bstring);
+ strcpy(h->outstring+h->opos+alen+1+strlen(h->separator), bstring);
+ //strcpy(h->outstring+h->opos+alen+blen+2,">");
+ strcpy(h->outstring+h->opos+alen+blen+1+strlen(h->separator),">");
+ //len = alen+blen+3;
+ len = alen+blen+2+strlen(h->separator);
+ }
+
+ else if (sym == IDENTITY) {
+ /* Apply up/down */
+ //idlen = utf8skip(h->instring+h->ipos)+1;
+ idlen = (h->sigmatch_array+h->ipos)->consumes; // here
+ strncpy(h->outstring+h->opos, h->instring+h->ipos, idlen);
+ strncpy(h->outstring+h->opos+idlen,"", 1);
+ len = idlen;
+ } else if (sym == EPSILON) {
+ return(0);
+ } else {
+ if (((h->mode) & DOWN) == DOWN) {
+ pstring = bstring;
+ len = blen;
+ } else {
+ pstring = astring;
+ len = alen;
+ }
+ memcpy(h->outstring+h->opos, pstring, len);
+ }
+ }
+ if (h->print_space && len > 0) {
+ strcpy(h->outstring+h->opos+len, h->space_symbol);
+ len++;
+ }
+ return(len);
+}
+
+int apply_match_length(struct apply_handle *h, int symbol) {
+ if (symbol == EPSILON) {
+ return 0;
+ }
+ if (h->has_flags && (h->flag_lookup+symbol)->type) {
+ return 0;
+ }
+ if (((h->mode) & ENUMERATE) == ENUMERATE) {
+ return 0;
+ }
+ if (h->ipos >= h->current_instring_length) {
+ return -1;
+ }
+ if ((h->sigmatch_array+(h->ipos))->signumber == symbol) {
+ return((h->sigmatch_array+(h->ipos))->consumes);
+ }
+ if ((symbol == IDENTITY) || (symbol == UNKNOWN)) {
+ if ((h->sigmatch_array+h->ipos)->signumber == IDENTITY) {
+ return((h->sigmatch_array+(h->ipos))->consumes);
+ }
+ }
+ return -1;
+}
+
+/* Match a symbol from sigma against the current position in string */
+/* Return the number of symbols consumed in input string */
+/* For flags, we consume 0 symbols of the input string, naturally */
+
+int apply_match_str(struct apply_handle *h, int symbol, int position) {
+ if (((h->mode) & ENUMERATE) == ENUMERATE) {
+ if (h->has_flags && (h->flag_lookup+symbol)->type) {
+ if (!h->obey_flags) {
+ return 0;
+ }
+ if (apply_check_flag(h,(h->flag_lookup+symbol)->type, (h->flag_lookup+symbol)->name, (h->flag_lookup+symbol)->value) == SUCCEED) {
+ return 0;
+ } else {
+ return -1;
+ }
+ }
+ return(0);
+ }
+
+
+ if (symbol == EPSILON) {
+ return 0;
+ }
+
+ /* If symbol is a flag, we need to check consistency */
+ if (h->has_flags && (h->flag_lookup+symbol)->type) {
+ if (!h->obey_flags) {
+ return 0;
+ }
+ if (apply_check_flag(h,(h->flag_lookup+symbol)->type, (h->flag_lookup+symbol)->name, (h->flag_lookup+symbol)->value) == SUCCEED) {
+ return 0;
+ } else {
+ return -1;
+ }
+ }
+
+ if (position >= h->current_instring_length) {
+ return -1;
+ }
+ if ((h->sigmatch_array+position)->signumber == symbol) {
+ return((h->sigmatch_array+position)->consumes);
+ }
+ if ((symbol == IDENTITY) || (symbol == UNKNOWN)) {
+ if ((h->sigmatch_array+position)->signumber == IDENTITY) {
+ return((h->sigmatch_array+position)->consumes);
+ }
+ }
+ return -1;
+}
+
+void apply_create_statemap(struct apply_handle *h, struct fsm *net) {
+ int i;
+ struct fsm_state *fsm;
+ fsm = net->states;
+ h->statemap = xxmalloc(sizeof(int)*net->statecount);
+ h->marks = xxmalloc(sizeof(int)*net->statecount);
+ h->numlines = xxmalloc(sizeof(int)*net->statecount);
+
+ for (i=0; i < net->statecount; i++) {
+ *(h->numlines+i) = 0; /* Only needed in binary search */
+ *(h->statemap+i) = -1;
+ *(h->marks+i) = 0;
+ }
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ *(h->numlines+(fsm+i)->state_no) = *(h->numlines+(fsm+i)->state_no)+1;
+ if (*(h->statemap+(fsm+i)->state_no) == -1) {
+ *(h->statemap+(fsm+i)->state_no) = i;
+ }
+ }
+}
+
+void apply_add_sigma_trie(struct apply_handle *h, int number, char *symbol, int len) {
+
+ /* Create a trie of sigma symbols (prefixes) so we can */
+ /* quickly (in O(n)) tokenize an arbitrary string into */
+ /* integer sequences representing symbols, using longest- */
+ /* leftmost factorization. */
+
+ int i;
+ struct sigma_trie *st;
+ struct sigma_trie_arrays *sta;
+
+ st = h->sigma_trie;
+ for (i = 0; i < len; i++) {
+ st = st+(unsigned char)*(symbol+i);
+ if (i == (len-1)) {
+ st->signum = number;
+ } else {
+ if (st->next == NULL) {
+ st->next = xxcalloc(256,sizeof(struct sigma_trie));
+ st = st->next;
+ /* store these arrays to free them later */
+ sta = xxmalloc(sizeof(struct sigma_trie_arrays));
+ sta->arr = st;
+ sta->next = h->sigma_trie_arrays;
+ h->sigma_trie_arrays = sta;
+ } else {
+ st = st->next;
+ }
+ }
+ }
+}
+
+void apply_mark_flagstates(struct apply_handle *h) {
+ int i;
+ struct fsm_state *fsm;
+
+ /* Create bitarray with those states that have a flag symbol on an arc */
+ /* This is needed to decide whether we can perform a binary search. */
+
+ if (!h->has_flags || h->flag_lookup == NULL) {
+ return;
+ }
+ if (h->flagstates) {
+ xxfree(h->flagstates);
+ }
+ h->flagstates = xxcalloc(BITNSLOTS(h->last_net->statecount), sizeof(uint8_t));
+ fsm = h->last_net->states;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target == -1) {
+ continue;
+ }
+ if ((h->flag_lookup+(fsm+i)->in)->type) {
+ BITSET(h->flagstates,(fsm+i)->state_no);
+ }
+ if ((h->flag_lookup+(fsm+i)->out)->type) {
+ BITSET(h->flagstates,(fsm+i)->state_no);
+ }
+ }
+}
+
+void apply_create_sigarray(struct apply_handle *h, struct fsm *net) {
+ struct sigma *sig;
+ int i, maxsigma;
+
+ maxsigma = sigma_max(net->sigma);
+ h->sigma_size = maxsigma+1;
+ // Default size created at init, resized later if necessary
+ h->sigmatch_array = xxcalloc(1024,sizeof(struct sigmatch_array));
+ h->sigmatch_array_size = 1024;
+
+ h->sigs = xxmalloc(sizeof(struct sigs)*(maxsigma+1));
+ h->has_flags = 0;
+ h->flag_list = NULL;
+
+ /* Malloc first array of trie and store trie ptrs to be able to free later */
+ /* when apply_clear() is called. */
+
+ h->sigma_trie = xxcalloc(256,sizeof(struct sigma_trie));
+ h->sigma_trie_arrays = xxmalloc(sizeof(struct sigma_trie_arrays));
+ h->sigma_trie_arrays->arr = h->sigma_trie;
+ h->sigma_trie_arrays->next = NULL;
+
+ for (i=0;i<256;i++)
+ (h->sigma_trie+i)->next = NULL;
+ for (sig = h->gsigma; sig != NULL && sig->number != -1; sig = sig->next) {
+ if (flag_check(sig->symbol)) {
+ h->has_flags = 1;
+ apply_add_flag(h, flag_get_name(sig->symbol));
+ }
+ (h->sigs+(sig->number))->symbol = sig->symbol;
+ (h->sigs+(sig->number))->length = strlen(sig->symbol);
+ /* Add sigma entry to trie */
+ if (sig->number > IDENTITY) {
+ apply_add_sigma_trie(h, sig->number, sig->symbol, (h->sigs+(sig->number))->length);
+ }
+ }
+ if (maxsigma >= IDENTITY) {
+ (h->sigs+EPSILON)->symbol = h->epsilon_symbol;
+ (h->sigs+EPSILON)->length = strlen(h->epsilon_symbol);
+ (h->sigs+UNKNOWN)->symbol = "?";
+ (h->sigs+UNKNOWN)->length = 1;
+ (h->sigs+IDENTITY)->symbol = "@";
+ (h->sigs+IDENTITY)->length = 1;
+ }
+ if (h->has_flags) {
+
+ h->flag_lookup = xxmalloc(sizeof(struct flag_lookup)*(maxsigma+1));
+ for (i=0; i <= maxsigma; i++) {
+ (h->flag_lookup+i)->type = 0;
+ (h->flag_lookup+i)->name = NULL;
+ (h->flag_lookup+i)->value = NULL;
+ }
+ for (sig = h->gsigma; sig != NULL ; sig = sig->next) {
+ if (flag_check(sig->symbol)) {
+ (h->flag_lookup+sig->number)->type = flag_get_type(sig->symbol);
+ (h->flag_lookup+sig->number)->name = flag_get_name(sig->symbol);
+ (h->flag_lookup+sig->number)->value = flag_get_value(sig->symbol);
+ }
+ }
+ apply_mark_flagstates(h);
+ }
+}
+
+/* We need to know which symbols in sigma we can match for all positions */
+/* in the input string. Alternatively, if there is no input string as is the case */
+/* when we just list the words or randomly search the graph, we can match */
+/* any symbol in sigma. */
+
+/* We create an array that for each position in the input string */
+/* has information on which symbol we can match at that position */
+/* as well as how many symbols matching consumes */
+
+void apply_create_sigmatch(struct apply_handle *h) {
+ char *symbol;
+ struct sigma_trie *st;
+ int i, j, inlen, lastmatch, consumes, cons;
+ /* We create a sigmatch array only in case we match against a real string */
+ if (((h->mode) & ENUMERATE) == ENUMERATE) {
+ return;
+ }
+ symbol = h->instring;
+ inlen = strlen(symbol);
+ h->current_instring_length = inlen;
+ if (inlen >= h->sigmatch_array_size) {
+ xxfree(h->sigmatch_array);
+ h->sigmatch_array = xxmalloc(sizeof(struct sigmatch_array)*(inlen));
+ h->sigmatch_array_size = inlen;
+ }
+ /* Find longest match in alphabet at current position */
+ /* by traversing the trie of alphabet symbols */
+ for (i=0; i < inlen; i += consumes ) {
+ st = h->sigma_trie;
+ for (j=0, lastmatch = 0; ; j++) {
+ if (*(symbol+i+j) == '\0') {
+ break;
+ }
+ st = st+(unsigned char)*(symbol+i+j);
+ if (st->signum != 0) {
+ lastmatch = st->signum;
+ if (st->next == NULL)
+ break;
+ st = st->next;
+ } else if (st->next != NULL) {
+ st = st->next;
+ } else {
+ break;
+ }
+ }
+ if (lastmatch != 0) {
+ (h->sigmatch_array+i)->signumber = lastmatch;
+ consumes = (h->sigs+lastmatch)->length;
+ } else {
+ /* Not found in trie */
+ (h->sigmatch_array+i)->signumber = IDENTITY;
+ consumes = utf8skip(symbol+i)+1;
+ }
+
+ /* If we now find trailing unicode combining characters (0300-036F): */
+ /* (1) Merge them all with current symbol */
+ /* (2) Declare the whole sequence one ? (IDENTITY) symbol */
+ /* Step 2 is motivated by the fact that */
+ /* if the input is S(symbol) + D(diacritic) */
+ /* and SD were a symbol in the alphabet, then this would have been */
+ /* found when searching the alphabet symbols earlier, so SD+ => ? */
+ /* Note that this means that a multi-char symbol followed by a */
+ /* diacritic gets converted to a single ?, e.g. */
+ /* [TAG] + D => ? if [TAG] is in the alphabet, but [TAG]+D isn't. */
+
+ for ( ; (cons = utf8iscombining((unsigned char *)(symbol+i+consumes))); consumes += cons) {
+ (h->sigmatch_array+i)->signumber = IDENTITY;
+ }
+ (h->sigmatch_array+i)->consumes = consumes;
+ }
+}
+
+void apply_add_flag(struct apply_handle *h, char *name) {
+ struct flag_list *flist, *flist_prev;
+ if (h->flag_list == NULL) {
+ flist = h->flag_list = xxmalloc(sizeof(struct flag_list));
+ } else {
+ for (flist = h->flag_list; flist != NULL; flist_prev = flist, flist = flist->next) {
+ if (strcmp(flist->name, name) == 0) {
+ return;
+ }
+ }
+ flist = xxmalloc(sizeof(struct flag_list));
+ flist_prev->next = flist;
+ }
+ flist->name = name;
+ flist->value = NULL;
+ flist->neg = 0;
+ flist->next = NULL;
+ return;
+}
+
+void apply_clear_flags(struct apply_handle *h) {
+ struct flag_list *flist;
+ for (flist = h->flag_list; flist != NULL; flist = flist->next) {
+ flist->value = NULL;
+ flist->neg = 0;
+ }
+ return;
+}
+
+/* Check for flag consistency by looking at the current states of */
+/* flags in flist */
+
+int apply_check_flag(struct apply_handle *h, int type, char *name, char *value) {
+ struct flag_list *flist, *flist2;
+ for (flist = h->flag_list; flist != NULL; flist = flist->next) {
+ if (strcmp(flist->name, name) == 0) {
+ break;
+ }
+ }
+ h->oldflagvalue = flist->value;
+ h->oldflagneg = flist->neg;
+
+ if (type == FLAG_UNIFY) {
+ if (flist->value == NULL) {
+ flist->value = xxstrdup(value);
+ return SUCCEED;
+ }
+ else if (strcmp(value, flist->value) == 0 && flist->neg == 0) {
+ return SUCCEED;
+ } else if (strcmp(value, flist->value) != 0 && flist->neg == 1) {
+ flist->value = xxstrdup(value);
+ flist->neg = 0;
+ return SUCCEED;
+ }
+ return FAIL;
+ }
+
+ if (type == FLAG_CLEAR) {
+ flist->value = NULL;
+ flist->neg = 0;
+ return SUCCEED;
+ }
+
+ if (type == FLAG_DISALLOW) {
+ if (flist->value == NULL) {
+ return SUCCEED;
+ }
+ if (value == NULL && flist->value != NULL) {
+ return FAIL;
+ }
+ if (strcmp(value, flist->value) != 0) {
+ if (flist->neg == 1)
+ return FAIL;
+ return SUCCEED;
+ }
+ if (strcmp(value, flist->value) == 0 && flist->neg == 1) {
+ return SUCCEED;
+ }
+ return FAIL;
+ }
+
+ if (type == FLAG_NEGATIVE) {
+ flist->value = value;
+ flist->neg = 1;
+ return SUCCEED;
+ }
+
+ if (type == FLAG_POSITIVE) {
+ flist->value = value;
+ flist->neg = 0;
+ return SUCCEED;
+ }
+
+ if (type == FLAG_REQUIRE) {
+
+ if (value == NULL) {
+ if (flist->value == NULL) {
+ return FAIL;
+ } else {
+ return SUCCEED;
+ }
+ } else {
+ if (flist->value == NULL) {
+ return FAIL;
+ }
+ if (strcmp(value, flist->value) != 0) {
+ return FAIL;
+ } else {
+ if (flist->neg == 1) {
+ return FAIL;
+ }
+ return SUCCEED;
+ }
+ }
+ }
+
+ if (type == FLAG_EQUAL) {
+ for (flist2 = h->flag_list; flist2 != NULL; flist2 = flist2->next) {
+ if (strcmp(flist2->name, value) == 0) {
+ break;
+ }
+ }
+
+ if (flist2 == NULL && flist->value != NULL)
+ return FAIL;
+ if (flist2 == NULL && flist->value == NULL) {
+ return SUCCEED;
+ }
+ if (flist2->value == NULL || flist->value == NULL) {
+ if (flist2->value == NULL && flist->value == NULL && flist->neg == flist2->neg) {
+ return SUCCEED;
+ } else {
+ return FAIL;
+ }
+ } else if (strcmp(flist2->value, flist->value) == 0 && flist->neg == flist2->neg) {
+ return SUCCEED;
+ }
+ return FAIL;
+ }
+ fprintf(stderr,"***Don't know what do with flag [%i][%s][%s]\n", type, name, value);
+ return FAIL;
+}
diff --git a/cgflookup.c b/cgflookup.c
new file mode 100644
index 0000000..634b715
--- /dev/null
+++ b/cgflookup.c
@@ -0,0 +1,303 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include <ctype.h>
+#include <stdio.h>
+#include <limits.h>
+#include <getopt.h>
+#include <unistd.h>
+#include <wctype.h>
+#include <locale.h>
+#include "fomalib.h"
+
+#define LINE_LIMIT 262144
+#define UDP_MAX 65535
+#define FLOOKUP_PORT 6062
+
+static char *usagestring = "Usage: cgflookup [-h] [-a] [-i] [-s \"separator\"] [-w \"wordseparator\"] [-v] [-x] [-b] [-I <#|#k|#m|f>] <binary foma file>\n";
+
+static char *helpstring =
+"Applies words from stdin to a foma transducer/automaton read from a file and prints results to stdout.\n"
+
+"If the file contains several nets, inputs will be passed through all of them (simulating composition) or applied as alternates if the -a flag is specified (simulating priority union: the first net is tried first, if that fails to produce an output, then the second is tried, etc.).\n\n"
+"Options:\n\n"
+"-h\t\tprint help\n"
+"-a\t\ttry alternatives (in order of nets loaded, default is to pass words through each)\n"
+"-b\t\tunbuffered output (flushes output after each input word, for use in bidirectional piping)\n"
+"-i\t\tinverse application (apply down instead of up)\n"
+"-I indextype\tindex arcs with indextype (one of -I f -I #k -I #m or -I #)\n"
+"\t\t(usually slower than the default except for states > 1,000 arcs)\n"
+"\t\t -I # will index all states containing # arcs or more\n"
+"\t\t -I NUMk will index states from densest to sparsest until reaching mem limit of # kB\n"
+"\t\t -I NUMM will index states from densest to sparsest until reaching mem limit of # MB\n"
+"\t\t -I f will index flag-containing states only\n"
+"-q\t\tdon't sort arcs before applying (usually slower, except for really small, sparse automata)\n"
+"-s \"separator\"\tchange input/output separator symbol (default is TAB)\n"
+"-u \"separator\"\tmark uppercase words with <*>\n"
+"-w \"separator\"\tchange words separator symbol (default is LF)\n"
+ "-v\t\tprint version number\n";
+
+struct lookup_chain {
+ struct fsm *net;
+ struct apply_handle *ah;
+ struct lookup_chain *next;
+ struct lookup_chain *prev;
+};
+
+#define DIR_DOWN 0
+#define DIR_UP 1
+
+static char buffer[2048];
+static int apply_alternates = 0, numnets = 0, direction = DIR_UP, results, buffered_output = 1, index_arcs = 0, index_flag_states = 0, index_cutoff = 0, index_mem_limit = INT_MAX, mark_uppercase = 0;
+static char *separator = "\t", *wordseparator = "", *line, *indent = "\t";
+static FILE *INFILE;
+static struct lookup_chain *chain_head, *chain_tail, *chain_new, *chain_pos;
+static fsm_read_binary_handle fsrh;
+
+static char *(*applyer)() = &apply_up; /* Default apply direction = up */
+static void handle_line(char *s);
+static void app_print(char *result);
+static char *get_next_line();
+
+void app_print(char *result) {
+ wchar_t testuc[1]; // Temp storage to test uc of first letter in string
+ if (result == NULL) {
+ fprintf(stdout, "\"<%s>\"\n", line);
+ } else {
+ /* Or format string first */
+ if (mark_uppercase) {
+ mbstowcs(testuc, line, 1);
+ if (iswupper(*testuc)) {
+ fprintf(stdout,"%s%s <*>\n",indent, result);
+ } else {
+ fprintf(stdout,"%s%s\n",indent, result);
+ }
+ } else {
+ fprintf(stdout,"%s%s\n",indent, result);
+ }
+ }
+}
+
+int main(int argc, char *argv[]) {
+ int opt, sortarcs = 1;
+ char *infilename;
+ struct fsm *net;
+
+ setvbuf(stdout, buffer, _IOFBF, sizeof(buffer));
+
+ while ((opt = getopt(argc, argv, "abhHiI:qs:uw:vx")) != -1) {
+ switch(opt) {
+ case 'a':
+ apply_alternates = 1;
+ break;
+ case 'b':
+ buffered_output = 0;
+ break;
+ case 'h':
+ printf("%s%s\n", usagestring,helpstring);
+ exit(0);
+ case 'i':
+ direction = DIR_DOWN;
+ applyer = &apply_down;
+ break;
+ case 'q':
+ sortarcs = 0;
+ break;
+ case 'I':
+ if (strcmp(optarg, "f") == 0) {
+ index_flag_states = 1;
+ index_arcs = 1;
+ } else if (strstr(optarg, "k") != NULL && strstr(optarg,"K") != NULL) {
+ /* k limit */
+ index_mem_limit = 1024*atoi(optarg);
+ index_arcs = 1;
+ } else if (strstr(optarg, "m") != NULL && strstr(optarg,"M") != NULL) {
+ /* m limit */
+ index_mem_limit = 1024*1024*atoi(optarg);
+ index_arcs = 1;
+ } else if (isdigit(*optarg)) {
+ index_arcs = 1;
+ index_cutoff = atoi(optarg);
+ }
+ break;
+ case 's':
+ separator = strdup(optarg);
+ break;
+ case 'u':
+ mark_uppercase = 1;
+ if (!setlocale(LC_CTYPE, "")) {
+ fprintf(stderr, "Check uppercase flag is on, but can't set locale!\n");
+ }
+ break;
+ case 'w':
+ wordseparator = strdup(optarg);
+ break;
+ case 'v':
+ printf("cgflookup 1.03 (foma library version %s)\n", fsm_get_library_version_string());
+ exit(0);
+ default:
+ fprintf(stderr, "%s", usagestring);
+ exit(EXIT_FAILURE);
+ }
+ }
+ if (optind == argc) {
+ fprintf(stderr, "%s", usagestring);
+ exit(EXIT_FAILURE);
+ }
+
+ infilename = argv[optind];
+
+ if ((fsrh = fsm_read_binary_file_multiple_init(infilename)) == NULL) {
+ perror("File error");
+ exit(EXIT_FAILURE);
+ }
+ chain_head = chain_tail = NULL;
+
+ while ((net = fsm_read_binary_file_multiple(fsrh)) != NULL) {
+ numnets++;
+ chain_new = xxmalloc(sizeof(struct lookup_chain));
+ if (direction == DIR_DOWN && net->arcs_sorted_in != 1 && sortarcs) {
+ fsm_sort_arcs(net, 1);
+ }
+ if (direction == DIR_UP && net->arcs_sorted_out != 1 && sortarcs) {
+ fsm_sort_arcs(net, 2);
+ }
+ chain_new->net = net;
+ chain_new->ah = apply_init(net);
+ if (direction == DIR_DOWN && index_arcs) {
+ apply_index(chain_new->ah, APPLY_INDEX_INPUT, index_cutoff, index_mem_limit, index_flag_states);
+ }
+ if (direction == DIR_UP && index_arcs) {
+ apply_index(chain_new->ah, APPLY_INDEX_OUTPUT, index_cutoff, index_mem_limit, index_flag_states);
+ }
+
+ chain_new->next = NULL;
+ chain_new->prev = NULL;
+ if (chain_tail == NULL) {
+ chain_tail = chain_head = chain_new;
+ } else if (direction == DIR_DOWN || apply_alternates == 1) {
+ chain_tail->next = chain_new;
+ chain_new->prev = chain_tail;
+ chain_tail = chain_new;
+ } else {
+ chain_new->next = chain_head;
+ chain_head->prev = chain_new;
+ chain_head = chain_new;
+ }
+ }
+
+ if (numnets < 1) {
+ fprintf(stderr, "%s: %s\n", "File error", infilename);
+ exit(EXIT_FAILURE);
+ }
+
+ /* Standard read from stdin */
+ line = xxcalloc(LINE_LIMIT, sizeof(char));
+ INFILE = stdin;
+ while (get_next_line() != NULL) {
+ results = 0;
+ handle_line(line);
+ if (results == 0) {
+ app_print(NULL);
+ }
+ fprintf(stdout, "%s", wordseparator);
+ if (!buffered_output) {
+ fflush(stdout);
+ }
+ }
+ /* Cleanup */
+ for (chain_pos = chain_head; chain_pos != NULL; chain_pos = chain_head) {
+ chain_head = chain_pos->next;
+ if (chain_pos->ah != NULL) {
+ apply_clear(chain_pos->ah);
+ }
+ if (chain_pos->net != NULL) {
+ fsm_destroy(chain_pos->net);
+ }
+ xxfree(chain_pos);
+ }
+ if (line != NULL)
+ xxfree(line);
+ exit(0);
+}
+
+char *get_next_line() {
+ char *r;
+ if ((r = fgets(line, LINE_LIMIT, INFILE)) != NULL) {
+ line[strcspn(line, "\n\r")] = '\0';
+ }
+ return r;
+}
+
+void handle_line(char *s) {
+ char *result, *tempstr;
+ /* Apply alternative */
+ results = 0;
+ if (apply_alternates == 1) {
+ for (chain_pos = chain_head, tempstr = s; ; chain_pos = chain_pos->next) {
+ result = applyer(chain_pos->ah, tempstr);
+ if (result != NULL) {
+ results++;
+ if (results == 1) {
+ fprintf(stdout, "\"<%s>\"\n",line);
+ }
+ app_print(result);
+ while ((result = applyer(chain_pos->ah, NULL)) != NULL) {
+ results++;
+ app_print(result);
+ }
+ break;
+ }
+ if (chain_pos == chain_tail) {
+ break;
+ }
+ }
+ } else {
+
+ /* Get result from chain */
+ for (chain_pos = chain_head, tempstr = s; ; chain_pos = chain_pos->next) {
+ result = applyer(chain_pos->ah, tempstr);
+ if (result != NULL && chain_pos != chain_tail) {
+ tempstr = result;
+ continue;
+ }
+ if (result != NULL && chain_pos == chain_tail) {
+ do {
+ results++;
+ if (results == 1) {
+ fprintf(stdout, "\"<%s>\"\n",line);
+ }
+ app_print(result);
+ } while ((result = applyer(chain_pos->ah, NULL)) != NULL);
+ }
+ if (result == NULL) {
+ /* Move up */
+ for (chain_pos = chain_pos->prev; chain_pos != NULL; chain_pos = chain_pos->prev) {
+ result = applyer(chain_pos->ah, NULL);
+ if (result != NULL) {
+ tempstr = result;
+ break;
+ }
+ }
+ }
+ if (chain_pos == NULL) {
+ break;
+ }
+ }
+ }
+}
diff --git a/cmatrix.l b/cmatrix.l
new file mode 100644
index 0000000..b2c7c53
--- /dev/null
+++ b/cmatrix.l
@@ -0,0 +1,97 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+%option noyywrap
+%option nounput
+%option noinput
+%{
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include "foma.h"
+
+extern int cmatrixlex();
+static struct fsm *mnet;
+static int currcost;
+static char *instring;
+
+void my_cmatrixparse(struct fsm *net, char *my_string) {
+
+ YY_BUFFER_STATE my_string_buffer;
+
+ currcost = 1;
+ my_string_buffer = cmatrix_scan_string(my_string);
+ mnet = net;
+ cmatrix_init(mnet);
+ cmatrixlex();
+ cmatrix_delete_buffer(my_string_buffer);
+}
+
+%}
+
+ANYUTF [\001-\177]|[\300-\337].|[\340-\357]..|[\360-\367]...
+NOCOLON ([\001-\177]{-}[\011\040\012\014\072])|[\300-\337].|[\340-\357]..|[\360-\367]...
+SP [\040]|[\011]|[\012]|[\014]
+
+%x SUB DEL INS COST OUTSTRING
+
+%%
+
+Substitute{SP}+/[0-9]+ { BEGIN(SUB); }
+Delete{SP}+/[0-9]+ { BEGIN(DEL); }
+Insert{SP}+/[0-9]+ { BEGIN(INS); }
+Cost{SP}+/[0-9]+ { BEGIN(COST); }
+^#.* { }
+
+:{NOCOLON}+ {
+ cmatrix_set_cost(mnet, NULL, cmatrixtext+1, currcost);
+}
+
+{NOCOLON}+: {
+ *(cmatrixtext+strlen(cmatrixtext)-1) = '\0';
+ cmatrix_set_cost(mnet, cmatrixtext, NULL, currcost);
+}
+
+{NOCOLON}+:/{NOCOLON}+ {
+ instring = xxstrndup(cmatrixtext, strlen(cmatrixtext)-1);
+ BEGIN(OUTSTRING);
+
+}
+<OUTSTRING>{NOCOLON}+ {
+ cmatrix_set_cost(mnet, instring, cmatrixtext, currcost);
+ BEGIN(INITIAL);
+}
+
+<SUB>[0-9]+ {
+ cmatrix_default_substitute(mnet, atoi(cmatrixtext));
+ BEGIN(INITIAL);
+}
+<DEL>[0-9]+ {
+ cmatrix_default_delete(mnet, atoi(cmatrixtext));
+ BEGIN(INITIAL);
+}
+<INS>[0-9]+ {
+ cmatrix_default_insert(mnet, atoi(cmatrixtext));
+ BEGIN(INITIAL);
+}
+
+<COST>[0-9]+ {
+ currcost = atoi(cmatrixtext);
+ BEGIN(INITIAL);
+}
+
+<*>[\012|\040] { }
diff --git a/coaccessible.c b/coaccessible.c
new file mode 100644
index 0000000..575acf9
--- /dev/null
+++ b/coaccessible.c
@@ -0,0 +1,176 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+#include "foma.h"
+
+struct invtable {
+ int state;
+ struct invtable *next;
+};
+
+struct fsm *fsm_coaccessible(struct fsm *net) {
+
+ struct invtable *inverses, *temp_i, *temp_i_prev, *current_ptr;
+ int i, j, s, t, *coacc, current_state, markcount, *mapping, terminate, new_linecount, new_arccount, *added, old_statecount;
+
+
+ struct fsm_state *fsm;
+
+ fsm = net->states;
+ new_arccount = 0;
+ /* printf("statecount %i\n",net->statecount); */
+ old_statecount = net->statecount;
+ inverses = xxcalloc(net->statecount, sizeof(struct invtable));
+ coacc = xxmalloc(sizeof(int)*(net->statecount));
+ mapping = xxmalloc(sizeof(int)*(net->statecount));
+ added = xxmalloc(sizeof(int)*(net->statecount));
+
+ for (i=0; i < (net->statecount); i++) {
+ (inverses+i)->state = -1;
+ *(coacc+i) = 0;
+ *(added+i) = 0;
+ }
+
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ s = (fsm+i)->state_no;
+ t = (fsm+i)->target;
+ if (t != -1 && s != t) {
+
+ if (((inverses+t)->state) == -1) {
+ (inverses+t)->state = s;
+ } else {
+ temp_i = xxmalloc(sizeof(struct invtable));
+ temp_i->next = (inverses+t)->next;
+ (inverses+t)->next = temp_i;
+ temp_i->state = s;
+ }
+ }
+ }
+
+ /* Push & mark finals */
+
+ markcount = 0;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->final_state && (!*(coacc+((fsm+i)->state_no)))) {
+ int_stack_push((fsm+i)->state_no);
+ *(coacc+(fsm+i)->state_no) = 1;
+ markcount++;
+ }
+ }
+
+ terminate = 0;
+ while(!int_stack_isempty()) {
+ current_state = int_stack_pop();
+ current_ptr = inverses+current_state;
+ while(current_ptr != NULL && current_ptr->state != -1) {
+ if (!*(coacc+(current_ptr->state))) {
+ *(coacc+(current_ptr->state)) = 1;
+ int_stack_push(current_ptr->state);
+ markcount++;
+ }
+ current_ptr = current_ptr->next;
+ }
+ if (markcount >= net->statecount) {
+ /* printf("Already coacc\n"); */
+ terminate = 1;
+ int_stack_clear();
+ break;
+ }
+ }
+
+
+ if (terminate == 0) {
+ *mapping = 0; /* state 0 always exists */
+ new_linecount = 0;
+ for (i=1,j=0; i < (net->statecount);i++) {
+ if (*(coacc+i) == 1) {
+ j++;
+ *(mapping+i) = j;
+ }
+ }
+
+ for (i=0,j=0; (fsm+i)->state_no != -1; i++) {
+ if (i > 0 && (fsm+i)->state_no != (fsm+i-1)->state_no && (fsm+i-1)->final_state && !*(added+((fsm+i-1)->state_no))) {
+ add_fsm_arc(fsm, j++, *(mapping+((fsm+i-1)->state_no)), -1, -1, -1, 1, (fsm+i-1)->start_state);
+ new_linecount++;
+ *(added+((fsm+i-1)->state_no)) = 1;
+ /* printf("addf ad %i\n",i); */
+ }
+ if (*(coacc+((fsm+i)->state_no)) && (((fsm+i)->target == -1) || *(coacc+((fsm+i)->target)))) {
+ (fsm+j)->state_no = *(mapping+((fsm+i)->state_no));
+ if ((fsm+i)->target == -1) {
+ (fsm+j)->target = -1;
+ } else {
+ (fsm+j)->target = *(mapping+((fsm+i)->target));
+ }
+ (fsm+j)->final_state = (fsm+i)->final_state;
+ (fsm+j)->start_state = (fsm+i)->start_state;
+ (fsm+j)->in = (fsm+i)->in;
+ (fsm+j)->out = (fsm+i)->out;
+ j++;
+ new_linecount++;
+ *(added+(fsm+i)->state_no) = 1;
+ if ((fsm+i)->target != -1) {
+ new_arccount++;
+ }
+ }
+ }
+
+ if ((i > 1) && ((fsm+i-1)->final_state) && *(added+((fsm+i-1)->state_no)) == 0) {
+ /* printf("addf\n"); */
+ add_fsm_arc(fsm, j++, *(mapping+((fsm+i-1)->state_no)), -1, -1, -1, 1, (fsm+i-1)->start_state);
+ new_linecount++;
+ }
+
+ if (new_linecount == 0) {
+ add_fsm_arc(fsm, j++, 0, -1, -1, -1, -1, -1);
+ }
+
+ add_fsm_arc(fsm, j, -1, -1, -1, -1, -1, -1);
+ if (markcount == 0) {
+ /* We're dealing with the empty language */
+ xxfree(fsm);
+ net->states = fsm_empty();
+ fsm_sigma_destroy(net->sigma);
+ net->sigma = sigma_create();
+ }
+ net->linecount = new_linecount;
+ net->arccount = new_arccount;
+ net->statecount = markcount;
+ }
+
+ /* printf("Markccount %i \n",markcount); */
+
+ for (i = 0; i < old_statecount ; i++) {
+ for (temp_i = inverses+i; temp_i != NULL ; ) {
+ temp_i_prev = temp_i;
+ temp_i = temp_i->next;
+ if (temp_i_prev != inverses+i)
+ xxfree(temp_i_prev);
+ }
+ }
+ xxfree(inverses);
+
+ xxfree(coacc);
+ xxfree(added);
+ xxfree(mapping);
+ net->is_pruned = YES;
+ return(net);
+}
diff --git a/constructions.c b/constructions.c
new file mode 100644
index 0000000..8dfe71a
--- /dev/null
+++ b/constructions.c
@@ -0,0 +1,3162 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "foma.h"
+
+#define KLEENE_STAR 0
+#define KLEENE_PLUS 1
+#define OPTIONALITY 2
+
+#define COMPLEMENT 0
+#define COMPLETE 1
+
+#define STACK_3_PUSH(a,b,c) int_stack_push(a); int_stack_push(b); int_stack_push(c);
+#define STACK_2_PUSH(a,b) int_stack_push(a); int_stack_push(b);
+
+struct mergesigma {
+ char *symbol;
+ unsigned char presence; /* 1 = in net 1, 2 = in net 2, 3 = in both */
+ int number;
+ struct mergesigma *next;
+};
+
+int sort_cmp(const void *a, const void *b) {
+ return (((const struct fsm_state *)a)->state_no - ((const struct fsm_state *)b)->state_no);
+}
+
+INLINE int add_fsm_arc(struct fsm_state *fsm, int offset, int state_no, int in, int out, int target, int final_state, int start_state);
+
+struct fsm *fsm_kleene_closure(struct fsm *net, int optionality);
+
+struct fsm *fsm_kleene_star(struct fsm *net) {
+ return (fsm_kleene_closure(net, KLEENE_STAR));
+}
+
+struct fsm *fsm_kleene_plus(struct fsm *net) {
+ return (fsm_kleene_closure(net, KLEENE_PLUS));
+}
+
+struct fsm *fsm_optionality(struct fsm *net) {
+ return (fsm_kleene_closure(net, OPTIONALITY));
+}
+
+struct fsm *fsm_escape(char *symbol) {
+ return(fsm_symbol(symbol+1));
+}
+
+/* Convert a multicharacter-string-containing machine */
+/* to the equivalent "letter" machine where all arcs */
+/* are single utf8 letters. */
+
+struct fsm *fsm_letter_machine(struct fsm *net) {
+
+ struct fsm *outnet;
+ struct fsm_read_handle *inh;
+ struct fsm_construct_handle *outh;
+ int i, steps, source, target, addstate, innum, outnum, inlen, outlen;
+ char *in, *out, *currin, *currout, tmpin[128], tmpout[128];
+
+ inh = fsm_read_init(fsm_minimize(net));
+ outh = fsm_construct_init("name");
+ addstate = net->statecount;
+
+ while (fsm_get_next_arc(inh)) {
+ in = fsm_get_arc_in(inh);
+ out = fsm_get_arc_out(inh);
+ innum = fsm_get_arc_num_in(inh);
+ outnum = fsm_get_arc_num_out(inh);
+ source = fsm_get_arc_source(inh);
+ target = fsm_get_arc_target(inh);
+
+ if (((innum > IDENTITY) && utf8strlen(in) > 1) || ((outnum > IDENTITY) && utf8strlen(out) > 1)) {
+ inlen = innum <= IDENTITY ? 1 : utf8strlen(in);
+ outlen = outnum <= IDENTITY ? 1 : utf8strlen(out);
+ steps = inlen > outlen ? inlen : outlen;
+
+ target = addstate;
+ for (i = 0; i < steps; i++) {
+ if (innum <= IDENTITY || inlen < 1) {
+ if (inlen < 1)
+ currin = "@_EPSILON_SYMBOL_@";
+ else
+ currin = in;
+ } else {
+ strncpy(tmpin, in, utf8skip(in)+1);
+ *(tmpin+utf8skip(in)+1) = '\0';
+ currin = tmpin;
+ inlen--;
+ in = in+utf8skip(in)+1;
+ }
+ if (outnum <= IDENTITY || outlen < 1) {
+ if (outlen < 1)
+ currout = "@_EPSILON_SYMBOL_@";
+ else
+ currout = out;
+ } else {
+ strncpy(tmpout, out, utf8skip(in)+1);
+ *(tmpout+utf8skip(out)+1) = '\0';
+ currout = tmpout;
+ out = out+utf8skip(out)+1;
+ outlen--;
+ }
+ if (i == 0 && steps > 1) {
+ target = addstate;
+ addstate++;
+ }
+ if (i > 0 && (steps-i == 1)) {
+ source = addstate - 1;
+ target = fsm_get_arc_target(inh);
+ }
+ if (i > 0 && (steps-i != 1)) {
+ source = addstate-1;
+ target = addstate;
+ addstate++;
+ }
+ fsm_construct_add_arc(outh,source,target,currin,currout);
+ }
+ } else {
+ fsm_construct_add_arc(outh,source,target,in,out);
+ }
+ }
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(outh, i);
+ }
+ while ((i = fsm_get_next_initial(inh)) != -1) {
+ fsm_construct_set_initial(outh, i);
+ }
+ fsm_read_done(inh);
+ outnet = fsm_construct_done(outh);
+ return(outnet);
+}
+
+struct fsm *fsm_explode(char *symbol) {
+ struct fsm *net;
+ struct fsm_construct_handle *h;
+ char *tempstring;
+ int length, i, j, skip;
+
+ h = fsm_construct_init("");
+ fsm_construct_set_initial(h,0);
+
+ length = strlen(symbol)-2;
+ for (i=1, j=1; i <= length; i += skip, j++) {
+ skip = utf8skip(symbol+i)+1;
+ tempstring = xxstrndup(((symbol+i)),skip);
+ fsm_construct_add_arc(h,j-1,j,tempstring,tempstring);
+ xxfree(tempstring);
+ }
+ fsm_construct_set_final(h, j-1);
+ net = fsm_construct_done(h);
+ return(net);
+}
+
+struct fsm *fsm_symbol(char *symbol) {
+ struct fsm *net;
+ int symbol_no;
+
+ net = fsm_create("");
+ fsm_update_flags(net, YES, YES, YES, YES, YES, NO);
+ if (strcmp(symbol,"@_EPSILON_SYMBOL_@")==0) {
+ /* Epsilon */
+ (void)sigma_add_special(EPSILON, net->sigma);
+ net->states = xxmalloc(sizeof(struct fsm_state)*2);
+ add_fsm_arc(net->states, 0, 0, -1,-1,-1,1,1);
+ add_fsm_arc(net->states, 1, -1,-1,-1,-1,-1,-1);
+ net->arccount = 0;
+ net->statecount = 1;
+ net->linecount = 1;
+ net->finalcount = 1;
+ net->is_deterministic = NO;
+ net->is_minimized = NO;
+ net->is_epsilon_free = NO;
+ } else {
+ if ((strcmp(symbol,"@_IDENTITY_SYMBOL_@") == 0)) {
+ symbol_no = sigma_add_special(IDENTITY,net->sigma);
+ } else {
+ symbol_no = sigma_add(symbol,net->sigma);
+ }
+ net->states = xxmalloc(sizeof(struct fsm_state)*3);
+ add_fsm_arc(net->states, 0, 0, symbol_no, symbol_no, 1, 0, 1);
+ add_fsm_arc(net->states, 1, 1, -1, -1, -1, 1, 0);
+ add_fsm_arc(net->states, 2, -1, -1, -1, -1, -1, -1);
+ net->arity = 1;
+ net->pathcount = 1;
+ net->arccount = 1;
+ net->statecount = 2;
+ net->linecount = 2;
+ net->finalcount = 1;
+ net->arcs_sorted_in = YES;
+ net->arcs_sorted_out = YES;
+ net->is_deterministic = YES;
+ net->is_minimized = YES;
+ net->is_epsilon_free = YES;
+ }
+ return(net);
+}
+
+void fsm_sort_lines(struct fsm *net) {
+ struct fsm_state *fsm;
+ fsm = net->states;
+ qsort(fsm, find_arccount(fsm), sizeof(struct fsm_state), sort_cmp);
+}
+
+void fsm_update_flags(struct fsm *net, int det, int pru, int min, int eps, int loop, int completed) {
+ net->is_deterministic = det;
+ net->is_pruned = pru;
+ net->is_minimized = min;
+ net->is_epsilon_free = eps;
+ net->is_loop_free = loop;
+ net->is_completed = completed;
+ net->arcs_sorted_in = NO;
+ net->arcs_sorted_out = NO;
+}
+
+int fsm_count_states(struct fsm_state *fsm) {
+ int i, temp = -1, states = 0;
+ for(i=0; (fsm+i)->state_no != -1; i++) {
+ if (temp != (fsm+i)->state_no) {
+ states++;
+ temp = (fsm+i)->state_no;
+ }
+ }
+ return(states);
+}
+
+struct state_arr {
+ int final;
+ int start;
+ struct fsm_state *transitions;
+};
+
+struct state_arr *init_state_pointers(struct fsm_state *fsm_state) {
+
+ /* Create an array for quick lookup of whether states are final, and a pointer to the first line regarding each state */
+
+ struct state_arr *state_arr;
+ int states, i, sold;
+ sold = -1;
+ states = fsm_count_states(fsm_state);
+ state_arr = xxmalloc(sizeof(struct state_arr)*(states+1));
+ for (i=0; i<states; i++) {
+ (state_arr+i)->final = 0;
+ (state_arr+i)->start = 0;
+ }
+
+ for (i=0; (fsm_state+i)->state_no != -1; i++) {
+ if ((fsm_state+i)->final_state == 1)
+ (state_arr+((fsm_state+i)->state_no))->final = 1;
+ if ((fsm_state+i)->start_state == 1)
+ (state_arr+((fsm_state+i)->state_no))->start = 1;
+ if ((fsm_state+i)->state_no != sold) {
+ (state_arr+((fsm_state+i)->state_no))->transitions = (fsm_state+i);
+ sold = (fsm_state+i)->state_no;
+ }
+ }
+ return(state_arr);
+}
+
+/* An open addressing scheme (with linear probing) to store triplets of ints */
+/* and hashing them with an automatically increasing key at every insert */
+/* The table is rehashed whenever occupancy reaches 0.5 */
+
+struct triplethash_triplets {
+ int a;
+ int b;
+ int c;
+ int key;
+};
+
+struct triplethash {
+ struct triplethash_triplets *triplets;
+ unsigned int tablesize;
+ int occupancy;
+};
+
+struct triplethash *triplet_hash_init() {
+ struct triplethash *th;
+ int i;
+ th = xxmalloc(sizeof(struct triplethash));
+ th->tablesize = 128;
+ th->occupancy = 0;
+ th->triplets = xxmalloc(sizeof(struct triplethash_triplets) * th->tablesize);
+ for (i = 0; i < th->tablesize; i++) {
+ (th->triplets+i)->key = -1;
+ }
+ return(th);
+}
+
+unsigned int triplethash_hashf(int a, int b, int c) {
+ return((unsigned int)(a * 7907 + b * 86028157 + c * 7919));
+}
+
+void triplet_hash_free(struct triplethash *th) {
+ if (th != NULL) {
+ if (th->triplets != NULL) {
+ xxfree(th->triplets);
+ }
+ xxfree(th);
+ }
+}
+
+void triplet_hash_rehash(struct triplethash *th);
+
+void triplet_hash_insert_with_key(struct triplethash *th, int a, int b, int c, int key) {
+ struct triplethash_triplets *th_table;
+ unsigned int hash;
+ th_table = th->triplets;
+ hash = triplethash_hashf(a, b, c) % th->tablesize;
+ for (;;) {
+ if ((th_table + hash)->key == -1) {
+ (th_table + hash)->key = key;
+ (th_table + hash)->a = a;
+ (th_table + hash)->b = b;
+ (th_table + hash)->c = c;
+ break;
+ }
+ hash++;
+ if (hash >= th->tablesize)
+ hash -= th->tablesize;
+ }
+}
+
+int triplet_hash_insert(struct triplethash *th, int a, int b, int c) {
+ struct triplethash_triplets *th_table;
+ unsigned int hash;
+ th_table = th->triplets;
+ hash = triplethash_hashf(a,b,c) % th->tablesize;
+ for (;;) {
+ if ((th_table + hash)->key == - 1) {
+ (th_table + hash)->key = th->occupancy;
+ (th_table + hash)->a = a;
+ (th_table + hash)->b = b;
+ (th_table + hash)->c = c;
+ th->occupancy = th->occupancy + 1;
+ if (th->occupancy > th->tablesize / 2) {
+ triplet_hash_rehash(th);
+ }
+ return(th->occupancy - 1);
+ }
+ hash++;
+ if (hash >= th->tablesize)
+ hash -= th->tablesize;
+ }
+}
+
+void triplet_hash_rehash(struct triplethash *th) {
+ int i;
+ unsigned int newtablesize, oldtablesize;
+ struct triplethash_triplets *oldtriplets;
+ newtablesize = th->tablesize * 2;
+ oldtablesize = th->tablesize;
+ oldtriplets = th->triplets;
+ th->triplets = xxmalloc(sizeof(struct triplethash_triplets) * newtablesize);
+ th->tablesize = newtablesize;
+ for (i = 0; i < newtablesize; i++) {
+ (th->triplets+i)->key = -1;
+ }
+ for (i = 0; i < oldtablesize; i++) {
+ if ((oldtriplets+i)-> key != -1) {
+ triplet_hash_insert_with_key(th, (oldtriplets+i)->a, (oldtriplets+i)->b, (oldtriplets+i)->c, (oldtriplets+i)->key);
+ }
+ }
+ xxfree(oldtriplets);
+}
+
+int triplet_hash_find(struct triplethash *th, int a, int b, int c) {
+ struct triplethash_triplets *th_table;
+ unsigned int hash, j;
+ th_table = th->triplets;
+ hash = triplethash_hashf(a, b, c) % th->tablesize;
+ for (j = 0; j < th->tablesize; j++) {
+ if ((th_table + hash)->key == - 1)
+ return -1;
+ if ((th_table + hash)->a == a && (th_table + hash)->b == b && (th_table + hash)->c == c) {
+ return((th_table + hash)->key);
+ }
+ hash++;
+ if (hash >= th->tablesize)
+ hash -= th->tablesize;
+ }
+ return -1;
+}
+
+struct fsm *fsm_intersect(struct fsm *net1, struct fsm *net2) {
+
+ struct blookup {int mainloop; int target; } *array, *bptr;
+ int a,b,current_state, current_start, current_final, target_number, sigma2size, mainloop;
+ struct fsm_state *machine_a, *machine_b;
+ struct state_arr *point_a, *point_b;
+ struct fsm *new_net;
+ struct triplethash *th;
+
+ net1 = fsm_minimize(net1);
+ net2 = fsm_minimize(net2);
+
+ if (fsm_isempty(net1) || fsm_isempty(net2)) {
+ fsm_destroy(net1);
+ fsm_destroy(net2);
+ return(fsm_empty_set());
+ }
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm_update_flags(net1, YES, NO, UNK, YES, UNK, UNK);
+
+ machine_a = net1->states;
+ machine_b = net2->states;
+
+ sigma2size = sigma_max(net2->sigma)+1;
+ array = xxcalloc(sigma2size*sigma2size, sizeof(struct blookup));
+ mainloop = 0;
+
+ /* Intersect two networks by the running-in-parallel method */
+ /* new state 0 = {0,0} */
+
+ STACK_2_PUSH(0,0);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 0, 0, 0);
+
+ fsm_state_init(sigma_max(net1->sigma));
+
+ point_a = init_state_pointers(machine_a);
+ point_b = init_state_pointers(machine_b);
+
+ while (!int_stack_isempty()) {
+
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ b = int_stack_pop();
+
+ current_state = triplet_hash_find(th, a, b, 0);
+ current_start = (((point_a+a)->start == 1) && ((point_b+b)->start == 1)) ? 1 : 0;
+ current_final = (((point_a+a)->final == 1) && ((point_b+b)->final == 1)) ? 1 : 0;
+
+ fsm_state_set_current_state(current_state, current_final, current_start);
+
+ /* Create a lookup index for machine b */
+ /* array[in][out] holds the target for this state and the symbol pair in:out */
+ /* Also, we keep track of whether an entry is fresh by the mainloop counter */
+ /* so we don't mistakenly use an old entry and don't have to clear the table */
+ /* between each state pair we encounter */
+
+ for (mainloop++, machine_b = (point_b+b)->transitions; machine_b->state_no == b; machine_b++) {
+ if (machine_b->in < 0) continue;
+ bptr = array+(machine_b->in*sigma2size)+machine_b->out;
+ bptr->mainloop = mainloop;
+ bptr->target = machine_b->target;
+ }
+
+ /* The main loop where we run the machines in parallel */
+ /* We look at each transition of a in this state, and consult the index of b */
+ /* we just created */
+
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ if (machine_a->in < 0 || machine_a->out < 0) continue;
+ bptr = array+(machine_a->in*sigma2size)+machine_a->out;
+
+ if (bptr->mainloop != mainloop)
+ continue;
+
+ if ((target_number = triplet_hash_find(th, machine_a->target, bptr->target, 0)) == -1) {
+ STACK_2_PUSH(bptr->target, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, bptr->target, 0);
+ }
+
+ fsm_state_add_arc(current_state, machine_a->in, machine_a->out, target_number, current_final, current_start);
+
+ }
+ fsm_state_end_state();
+ }
+ new_net = fsm_create("");
+ fsm_sigma_destroy(new_net->sigma);
+ new_net->sigma = net1->sigma;
+ net1->sigma = NULL;
+ fsm_destroy(net2);
+ fsm_destroy(net1);
+ fsm_state_close(new_net);
+ xxfree(point_a);
+ xxfree(point_b);
+ xxfree(array);
+ triplet_hash_free(th);
+ return(fsm_coaccessible(new_net));
+}
+
+struct fsm *fsm_compose(struct fsm *net1, struct fsm *net2) {
+
+
+ /* The composition algorithm is the basic naive composition where we lazily */
+ /* take the cross-product of states P and Q and move to a new state with symbols */
+ /* ain, bout if the symbols aout = bin. Also, if aout = 0 state p goes to */
+ /* its target, while q stays. Similarly, if bin = 0, q goes to its target */
+ /* while p stays. */
+
+ /* We have two variants of the algorithm to avoid creating multiple paths: */
+ /* 1) Bistate composition. In this variant, when we create a new state, we call it */
+ /* (p,q,mode) where mode = 0 or 1, depending on what kind of an arc we followed */
+ /* to get there. If we followed an x:y arc where x and y are both real symbols */
+ /* we always go to mode 0, however, if we followed an 0:y arc, we go to mode 1. */
+ /* from mode 1, we do not follow x:0 arcs. Each (p,q,mode) is unique, and */
+ /* from (p,q,X) we always consider the transitions from p and q. */
+ /* We never create arcs (x:0 0:y) yielding x:y. */
+
+ /* 2) Tristate composition. Here we always go to mode 0 with a x:y arc. */
+ /* (x:0,0:y) yielding x:y is allowed, but only in mode 0 */
+ /* (x:y y:z) is always allowed and results in target = mode 0 */
+ /* 0:y arcs lead to mode 2, and from there we stay in mode 2 with 0:y */
+ /* in mode 2 we only consider 0:y and x:y arcs */
+ /* x:0 arcs lead to mode 1, and from there we stay in mode 1 with x:0 */
+ /* in mode 1 we only consider x:0 and x:y arcs */
+
+ /* It seems unsettled which type of composition is better. Tristate is similar to */
+ /* the filter transducer given in Mohri, Pereira and Riley (1996) and works well */
+ /* for cases such as [a:0 b:0 c:0 .o. 0:d 0:e 0:f], yielding the shortest path. */
+ /* However, for generic cases, bistate seems to yield smaller transducers. */
+ /* The global variable g_compose_tristate is set to OFF by default */
+
+ struct outarray {
+ short int symin;
+ short int symout;
+ int target;
+ int mainloop;
+ } *outarray, *iptr, *currtail;
+
+ struct index {
+ struct outarray *tail;
+ } *index;
+
+ extern int g_compose_tristate, g_flag_is_epsilon;
+ int a,b,i,mainloop,current_state, current_start, current_final, target_number, ain, bin, aout, bout, asearch, max2sigma;
+ struct fsm_state *machine_a, *machine_b;
+ struct state_arr *point_a, *point_b;
+ struct triplethash *th;
+ int mode;
+ _Bool *is_flag = NULL;
+
+
+ net1 = fsm_minimize(net1);
+ net2 = fsm_minimize(net2);
+
+ if (fsm_isempty(net1) || fsm_isempty(net2)) {
+ fsm_destroy(net1);
+ fsm_destroy(net2);
+ return(fsm_empty_set());
+ }
+
+ /* If flag-is-epsilon is on, we need to add the flag symbols */
+ /* in both networks to each other's sigma so that UNKNOWN */
+ /* or IDENTITY symbols do not match these flags, since they are */
+ /* supposed to have the behavior of EPSILON */
+ /* And we need to do this before merging the sigmas, of course */
+
+ if (g_flag_is_epsilon) {
+ struct sigma *sig1, *sig2;
+ int flags1, flags2;
+ flags1 = flags2 = 0;
+ sig2 = net2->sigma;
+ max2sigma = sigma_max(net2->sigma);
+ for (sig1 = net1->sigma; sig1 != NULL; sig1 = sig1->next) {
+ if (flag_check(sig1->symbol)) {
+ flags1 = 1;
+ if (sigma_find(sig1->symbol, sig2) == -1) {
+ sigma_add(sig1->symbol, sig2);
+ }
+ }
+ }
+
+ sig1 = net1->sigma;
+ for (sig2 = net2->sigma; sig2 != NULL ; sig2 = sig2->next) {
+ if (flag_check(sig2->symbol)) {
+ if (sig2->number <= max2sigma) {
+ flags2 = 1;
+ }
+ if (sigma_find(sig2->symbol, sig1) == -1) {
+ sigma_add(sig2->symbol, sig1);
+ }
+ }
+ }
+ sigma_sort(net2);
+ sigma_sort(net1);
+ if (flags1 && flags2) {
+ printf("***Warning: flag-is-epsilon is ON and both networks contain flags in composition. This may yield incorrect results. Set flag-is-epsilon to OFF.\n");
+ }
+ }
+
+ fsm_merge_sigma(net1, net2);
+
+ if (g_flag_is_epsilon) {
+ /* Create lookup table for quickly checking if a symbol is a flag */
+ struct sigma *sig1;
+ is_flag = xxmalloc(sizeof(_Bool)*(sigma_max(net1->sigma)+1));
+ for (sig1 = net1->sigma; sig1 != NULL; sig1=sig1->next) {
+ if (flag_check(sig1->symbol)) {
+ *(is_flag+(sig1->number)) = 1;
+ } else {
+ *(is_flag+(sig1->number)) = 0;
+ }
+ }
+ }
+
+ fsm_update_flags(net1, YES, NO, UNK, YES, UNK, UNK);
+
+ machine_a = net1->states;
+ machine_b = net2->states;
+
+ max2sigma = sigma_max(net2->sigma);
+
+ /* Create an index for looking up input symbols in machine b quickly */
+ /* We store each machine_b->in symbol in outarray[symin][...] */
+ /* the array index[symin] points to the tail of the current list in outarray */
+ /* (we may have many entries for one input symbol as there may be many outputs */
+ /* The field mainloop tells us whether the entry is current as we want to loop */
+ /* UNKNOWN and IDENTITY are indexed as UNKNOWN because we need to find both */
+ /* as they share some semantics */
+
+ index = xxcalloc(max2sigma+1, sizeof(struct index));
+ outarray = xxcalloc((max2sigma+2)*(max2sigma+1), sizeof(struct outarray));
+
+ for (i=0; i <= max2sigma; i++) {
+ (index+i)->tail = outarray + ((max2sigma+2)*i);
+ }
+
+
+ /* Mode, a, b */
+ STACK_3_PUSH(0,0,0);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 0, 0, 0);
+
+ fsm_state_init(sigma_max(net1->sigma));
+
+ point_a = init_state_pointers(machine_a);
+ point_b = init_state_pointers(machine_b);
+
+ mainloop = 0;
+
+ while (!int_stack_isempty()) {
+
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ b = int_stack_pop();
+ mode = int_stack_pop();
+
+ current_state = triplet_hash_find(th, a,b,mode);
+ current_start = (((point_a+a)->start == 1) && ((point_b+b)->start == 1) && (mode == 0)) ? 1 : 0;
+ current_final = (((point_a+a)->final == 1) && ((point_b+b)->final == 1)) ? 1 : 0;
+
+ fsm_state_set_current_state(current_state, current_final, current_start);
+
+ /* Create the index for machine b in this state */
+ for (mainloop++, machine_b = (point_b+b)->transitions; machine_b->state_no == b ; machine_b++) {
+ int bindex;
+ /* Index b */
+ bindex = (machine_b->in == IDENTITY) ? UNKNOWN : machine_b->in;
+ if (bindex < 0 || machine_b->target < 0)
+ continue;
+
+ iptr = (index+bindex)->tail;
+ if (iptr->mainloop != mainloop) {
+ iptr = (index+bindex)->tail = outarray+(bindex*(max2sigma+2));
+ } else {
+ iptr++;
+ }
+ iptr->symin = machine_b->in;
+ iptr->symout = machine_b->out;
+ iptr->mainloop = mainloop;
+ iptr->target = machine_b->target;
+ (index+bindex)->tail = iptr;
+ }
+
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+
+ /* If we have the same transition from (a,b)-> some state */
+ /* If we have x:y y:z trans to some state */
+ aout = machine_a->out;
+ ain = machine_a->in;
+ /* IDENTITY is indexed under UNKNOWN (see above) */
+ asearch = (aout == IDENTITY) ? UNKNOWN : aout;
+ if (aout < 0) continue;
+ iptr = outarray+(asearch*(max2sigma+2));
+ currtail = (index+asearch)->tail + 1;
+ for ( ; iptr != currtail && iptr->mainloop == mainloop ; iptr++) {
+
+ bin = iptr->symin;
+ bout = iptr->symout;
+
+ if (aout == IDENTITY && bin == UNKNOWN) {
+ ain = aout = UNKNOWN;
+ }
+ else if (aout == UNKNOWN && bin == IDENTITY) {
+ bin = bout = UNKNOWN;
+ }
+
+ if (!g_compose_tristate) {
+ if (bin == aout && bin != -1 && bin != EPSILON) {
+ /* mode -> 0 */
+ if ((target_number = triplet_hash_find(th, machine_a->target, iptr->target, 0)) == -1) {
+ STACK_3_PUSH(0, iptr->target, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, iptr->target, 0);
+ }
+
+ fsm_state_add_arc(current_state, ain, bout, target_number, current_final, current_start);
+ }
+ }
+
+ else if (g_compose_tristate) {
+ if (bin == aout && bin != -1 && ((bin != EPSILON || mode == 0))) {
+ /* mode -> 0 */
+ if ((target_number = triplet_hash_find(th, machine_a->target, iptr->target, 0)) == -1) {
+ STACK_3_PUSH(0, iptr->target, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, iptr->target, 0);
+ }
+
+ fsm_state_add_arc(current_state, ain, bout, target_number, current_final, current_start);
+ }
+ }
+
+ }
+ }
+
+ /* Treat epsilon outputs on machine a (may include flags) */
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ aout = machine_a->out;
+ if (aout != EPSILON && g_flag_is_epsilon == 0)
+ continue;
+ ain = machine_a->in;
+
+ if (g_flag_is_epsilon && aout != -1 && mode == 0 && *(is_flag+aout)) {
+ if ((target_number = triplet_hash_find(th, machine_a->target, b, 0)) == -1) {
+ STACK_3_PUSH(0, b, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, b, 0);
+ }
+ fsm_state_add_arc(current_state, ain, aout, target_number, current_final, current_start);
+ }
+
+ if (!g_compose_tristate) {
+ /* Check A:0 arcs on upper side */
+ if (aout == EPSILON && mode == 0) {
+ /* mode -> 0 */
+ if ((target_number = triplet_hash_find(th, machine_a->target, b, 0)) == -1) {
+ STACK_3_PUSH(0, b, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, b, 0);
+ }
+
+ fsm_state_add_arc(current_state, ain, EPSILON, target_number, current_final, current_start);
+ }
+ }
+
+ else if (g_compose_tristate) {
+ if (aout == EPSILON && (mode != 2)) {
+ /* mode -> 1 */
+ if ((target_number = triplet_hash_find(th, machine_a->target, b, 1)) == -1) {
+ STACK_3_PUSH(1, b, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, b, 1);
+ }
+
+ fsm_state_add_arc(current_state, ain, EPSILON, target_number, current_final, current_start);
+
+ }
+ }
+
+ }
+ /* Treat epsilon inputs on machine b (may include flags) */
+ for (machine_b = (point_b+b)->transitions; machine_b->state_no == b ; machine_b++) {
+ bin = machine_b->in;
+ if (bin != EPSILON && g_flag_is_epsilon == 0)
+ continue;
+
+ bout = machine_b->out;
+
+ if (g_flag_is_epsilon && bin != -1 && *(is_flag+bin)) {
+ if ((target_number = triplet_hash_find(th, a, machine_b->target, 1)) == -1) {
+ STACK_3_PUSH(1, machine_b->target,a);
+ target_number = triplet_hash_insert(th, a, machine_b->target, 1);
+ }
+ fsm_state_add_arc(current_state, bin, bout, target_number, current_final, current_start);
+ }
+
+ if (!g_compose_tristate) {
+ /* Check 0:A arcs on lower side */
+ if (bin == EPSILON) {
+ /* mode -> 1 */
+ if ((target_number = triplet_hash_find(th, a, machine_b->target, 1)) == -1) {
+ STACK_3_PUSH(1, machine_b->target,a);
+ target_number = triplet_hash_insert(th, a, machine_b->target, 1);
+ }
+
+ fsm_state_add_arc(current_state, EPSILON, bout, target_number, current_final, current_start);
+ }
+ }
+
+ else if (g_compose_tristate) {
+ /* Check 0:A arcs on lower side */
+ if (bin == EPSILON && mode != 1) {
+ /* mode -> 1 */
+ if ((target_number = triplet_hash_find(th, a, machine_b->target, 2)) == -1) {
+ STACK_3_PUSH(2, machine_b->target, a);
+ target_number = triplet_hash_insert(th, a, machine_b->target, 2);
+ }
+
+ fsm_state_add_arc(current_state, EPSILON, bout, target_number, current_final, current_start);
+ }
+ }
+ }
+ fsm_state_end_state();
+ }
+
+ xxfree(net1->states);
+ fsm_destroy(net2);
+ fsm_state_close(net1);
+ xxfree(point_a);
+ xxfree(point_b);
+ xxfree(index);
+ xxfree(outarray);
+
+ if (g_flag_is_epsilon)
+ xxfree(is_flag);
+ triplet_hash_free(th);
+ net1 = fsm_topsort(fsm_coaccessible(net1));
+ return(fsm_coaccessible(net1));
+}
+
+struct mergesigma *add_to_mergesigma(struct mergesigma *msigma, struct sigma *sigma, short presence) {
+ int number = 0;
+
+ if (msigma->number == -1) {
+ number = 2;
+ } else {
+ msigma->next = xxmalloc(sizeof(struct mergesigma));
+ number = msigma->number;
+ msigma = msigma->next;
+ msigma->next = NULL;
+ }
+
+ if (sigma->number < 3) {
+ msigma->number = sigma->number;
+ } else {
+ if (number < 3)
+ number = 2;
+ msigma->number = number+1;
+ }
+ msigma->symbol = sigma->symbol;
+ msigma->presence = presence;
+ return(msigma);
+}
+
+struct sigma *copy_mergesigma(struct mergesigma *mergesigma) {
+ struct sigma *sigma, *new_sigma;
+
+ sigma = new_sigma = NULL;
+ while(mergesigma != NULL) {
+ if (sigma == NULL) {
+ sigma = xxmalloc(sizeof(struct sigma));
+ new_sigma = sigma;
+ } else {
+ sigma->next = xxmalloc(sizeof(struct sigma));
+ sigma = sigma->next;
+ }
+ sigma->next = NULL;
+ sigma->number = mergesigma->number;
+
+ sigma->symbol = NULL;
+ if (mergesigma->symbol != NULL)
+ sigma->symbol = xxstrdup(mergesigma->symbol);
+ mergesigma = mergesigma->next;
+ }
+ return(new_sigma);
+}
+
+void fsm_merge_sigma(struct fsm *net1, struct fsm *net2) {
+
+ struct sigma *sigma_1, *sigma_2, *new_sigma_1 = NULL, *new_sigma_2 = NULL;
+ struct mergesigma *mergesigma, *mergesigma2, *start_mergesigma;
+ struct fsm_state *fsm_state, *new_1_state, *new_2_state;
+ int i, j, end_1 = 0, end_2 = 0, sigmasizes, *mapping_1, *mapping_2, equal = 1, unknown_1 = 0, unknown_2 = 0, net_unk = 0, net_adds = 0, net_lines;
+
+ i = sigma_find(".#.", net1->sigma);
+ j = sigma_find(".#.", net2->sigma);
+ if (i != -1 && j == -1) {
+ sigma_add(".#.", net2->sigma);
+ sigma_sort(net2);
+ }
+ if (j != -1 && i == -1) {
+ sigma_add(".#.", net1->sigma);
+ sigma_sort(net1);
+ }
+
+ sigma_1 = net1->sigma;
+ sigma_2 = net2->sigma;
+
+ sigmasizes = sigma_size(sigma_1) + sigma_size(sigma_2);
+
+ mapping_1 = xxmalloc(sizeof(int)*(sigmasizes+3));
+ mapping_2 = xxmalloc(sizeof(int)*(sigmasizes+3));
+
+ /* Fill mergesigma */
+
+ mergesigma = xxmalloc(sizeof(struct mergesigma));
+ mergesigma->number = -1;
+ mergesigma->symbol = NULL;
+ mergesigma->next = NULL;
+ start_mergesigma = mergesigma;
+
+ /* Loop over sigma 1, sigma 2 */
+ for (;;) {
+ if (sigma_1 == NULL)
+ end_1 = 1;
+ if (sigma_2 == NULL)
+ end_2 = 1;
+ if (end_1 && end_2)
+ break;
+ if (end_2) {
+ /* Treating only 1 now */
+ mergesigma = add_to_mergesigma(mergesigma, sigma_1, 1);
+ *(mapping_1+(sigma_1->number)) = mergesigma->number;
+ sigma_1 = sigma_1->next;
+ equal = 0;
+ continue;
+ }
+ else if (end_1) {
+ /* Treating only 2 now */
+ mergesigma = add_to_mergesigma(mergesigma, sigma_2, 2);
+ *(mapping_2+(sigma_2->number)) = mergesigma->number;
+ sigma_2 = sigma_2->next;
+ equal = 0;
+ continue;
+ }
+
+ else {
+
+ /* Both alive */
+
+ /* 1 or 2 contains special characters */
+ if ((sigma_1->number <= IDENTITY) || (sigma_2->number <= IDENTITY)) {
+
+ /* Treating zeros or unknowns */
+
+ if ((sigma_1->number == UNKNOWN) || (sigma_1->number == IDENTITY))
+ unknown_1 = 1;
+ if ((sigma_2->number == UNKNOWN) || (sigma_2->number == IDENTITY))
+ unknown_2 = 1;
+
+ if (sigma_1->number == sigma_2->number) {
+ mergesigma = add_to_mergesigma(mergesigma, sigma_1, 3);
+ sigma_1 = sigma_1->next;
+ sigma_2 = sigma_2->next;
+ }
+ else if (sigma_1->number < sigma_2->number) {
+ mergesigma = add_to_mergesigma(mergesigma, sigma_1, 1);
+ sigma_1 = sigma_1->next;
+ equal = 0;
+ }
+ else {
+ mergesigma = add_to_mergesigma(mergesigma, sigma_2, 2);
+ sigma_2 = sigma_2->next;
+ equal = 0;
+ }
+ continue;
+ }
+ /* Both contain non-special chars */
+ if (strcmp(sigma_1->symbol, sigma_2->symbol) == 0) {
+ mergesigma = add_to_mergesigma(mergesigma, sigma_1, 3);
+ /* Add symbol numbers to mapping */
+ *(mapping_1+(sigma_1->number)) = mergesigma->number;
+ *(mapping_2+(sigma_2->number)) = mergesigma->number;
+
+ sigma_1 = sigma_1->next;
+ sigma_2 = sigma_2->next;
+ }
+ else if (strcmp(sigma_1->symbol, sigma_2->symbol) < 0) {
+ mergesigma = add_to_mergesigma(mergesigma, sigma_1, 1);
+ *(mapping_1+(sigma_1->number)) = mergesigma->number;
+ sigma_1 = sigma_1->next;
+ equal = 0;
+ }
+ else {
+ mergesigma = add_to_mergesigma(mergesigma, sigma_2, 2);
+ *(mapping_2+(sigma_2->number)) = mergesigma->number;
+ sigma_2 = sigma_2->next;
+ equal = 0;
+ }
+ continue;
+ }
+ }
+
+ /* Go over both net1 and net2 and replace arc numbers with new mappings */
+
+ fsm_state = net1->states;
+ for (i=0; (fsm_state+i)->state_no != -1; i++) {
+ if ((fsm_state+i)->in > 2)
+ (fsm_state+i)->in = *(mapping_1+(fsm_state+i)->in);
+ if ((fsm_state+i)->out > 2)
+ (fsm_state+i)->out = *(mapping_1+(fsm_state+i)->out);
+ }
+ fsm_state = net2->states;
+ for (i=0; (fsm_state+i)->state_no != -1; i++) {
+ if ((fsm_state+i)->in > 2)
+ (fsm_state+i)->in = *(mapping_2+(fsm_state+i)->in);
+ if ((fsm_state+i)->out > 2)
+ (fsm_state+i)->out = *(mapping_2+(fsm_state+i)->out);
+ }
+
+ /* Copy mergesigma to net1, net2 */
+
+ new_sigma_1 = copy_mergesigma(start_mergesigma);
+ new_sigma_2 = copy_mergesigma(start_mergesigma);
+
+ fsm_sigma_destroy(net1->sigma);
+ fsm_sigma_destroy(net2->sigma);
+
+ net1->sigma = new_sigma_1;
+ net2->sigma = new_sigma_2;
+
+ /* Expand on ?, ?:x, y:? */
+
+ if (unknown_1 && !equal) {
+ /* Expand net 1 */
+ fsm_state = net1->states;
+ net_lines = find_arccount(net1->states);
+ for(mergesigma = start_mergesigma; mergesigma != NULL; mergesigma=mergesigma->next) {
+ if(mergesigma->presence == 2) {
+ net_unk++;
+ }
+ }
+ for(net_adds = 0, i=0; (fsm_state+i)->state_no != -1; i++) {
+ if ((fsm_state+i)->in == IDENTITY)
+ net_adds += net_unk;
+ if (((fsm_state+i)->in == UNKNOWN) && ((fsm_state+i)->out != UNKNOWN))
+ net_adds += net_unk;
+ if (((fsm_state+i)->out == UNKNOWN) && ((fsm_state+i)->in != UNKNOWN))
+ net_adds += net_unk;
+ if (((fsm_state+i)->in == UNKNOWN) && ((fsm_state+i)->out == UNKNOWN))
+ net_adds += net_unk*net_unk - net_unk + 2*net_unk;
+ }
+
+ new_1_state = xxmalloc(sizeof(struct fsm_state)*(net_adds+net_lines+1));
+ for(i=0,j=0; (fsm_state+i)->state_no != -1; i++) {
+
+ if ((fsm_state+i)->in == IDENTITY) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma=start_mergesigma; mergesigma != NULL; mergesigma=mergesigma->next) {
+ if ((mergesigma->presence == 2) && (mergesigma->number > IDENTITY)) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, mergesigma->number, mergesigma->number, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+
+ if ((fsm_state+i)->in == UNKNOWN && (fsm_state+i)->out != UNKNOWN) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma=start_mergesigma; mergesigma!=NULL; mergesigma=mergesigma->next) {
+ if ((mergesigma->presence == 2) && (mergesigma->number > IDENTITY)) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, mergesigma->number, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+
+ if (((fsm_state+i)->in != UNKNOWN) && ((fsm_state+i)->out == UNKNOWN)) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma=start_mergesigma; mergesigma != NULL; mergesigma = mergesigma->next) {
+ if ((mergesigma->presence == 2) && (mergesigma->number > IDENTITY)) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, mergesigma->number, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+
+ /* Replace ?:? with ?:[all unknowns] [all unknowns]:? and [all unknowns]:[all unknowns] where a != b */
+ if ((fsm_state+i)->in == UNKNOWN && (fsm_state+i)->out == UNKNOWN) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma2=start_mergesigma; mergesigma2 != NULL ; mergesigma2 = mergesigma2->next) {
+ for (mergesigma=start_mergesigma; mergesigma!=NULL; mergesigma=mergesigma->next) {
+ if (((mergesigma->presence == 2 && mergesigma2->presence == 2 && mergesigma->number > IDENTITY && mergesigma2->number > IDENTITY) || (mergesigma->number == UNKNOWN && mergesigma2->number > IDENTITY && mergesigma2->presence == 2) || (mergesigma2->number == UNKNOWN && mergesigma->number > IDENTITY && mergesigma->presence == 2)) && mergesigma->number != mergesigma2->number) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, mergesigma->number, mergesigma2->number, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+ }
+
+ /* Simply copy arcs that are not IDENTITY or UNKNOWN */
+ if (((fsm_state+i)->in > IDENTITY || (fsm_state+i)->in == EPSILON) && ((fsm_state+i)->out > IDENTITY || (fsm_state+i)->out == EPSILON)) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+
+ if ((fsm_state+i)->in == -1) {
+ add_fsm_arc(new_1_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+
+ add_fsm_arc(new_1_state, j, -1, -1, -1, -1, -1, -1);
+ xxfree(net1->states);
+ net1->states = new_1_state;
+ }
+
+ if (unknown_2 && !equal) {
+ /* Expand net 2 */
+ fsm_state = net2->states;
+ net_lines = find_arccount(net2->states);
+ for(net_unk = 0, mergesigma = start_mergesigma; mergesigma != NULL; mergesigma=mergesigma->next) {
+ if(mergesigma->presence == 1) {
+ net_unk++;
+ }
+ }
+
+ for(net_adds = 0, i=0; (fsm_state+i)->state_no != -1; i++) {
+ if ((fsm_state+i)->in == IDENTITY)
+ net_adds += net_unk;
+ if (((fsm_state+i)->in == UNKNOWN) && ((fsm_state+i)->out != UNKNOWN))
+ net_adds += net_unk;
+ if (((fsm_state+i)->out == UNKNOWN) && ((fsm_state+i)->in != UNKNOWN))
+ net_adds += net_unk;
+ if (((fsm_state+i)->in == UNKNOWN) && ((fsm_state+i)->out == UNKNOWN))
+ net_adds += net_unk*net_unk - net_unk + 2*net_unk;
+ }
+
+ /* We need net_add new lines in fsm_state */
+ new_2_state = xxmalloc(sizeof(struct fsm_state)*(net_adds+net_lines+1));
+ for(i=0,j=0; (fsm_state+i)->state_no != -1; i++) {
+
+ if ((fsm_state+i)->in == IDENTITY) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma=start_mergesigma; mergesigma!=NULL; mergesigma=mergesigma->next) {
+ if ((mergesigma->presence == 1) && (mergesigma->number > IDENTITY)) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, mergesigma->number, mergesigma->number, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+
+ if ((fsm_state+i)->in == UNKNOWN && (fsm_state+i)->out != UNKNOWN) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma=start_mergesigma; mergesigma!=NULL; mergesigma=mergesigma->next) {
+ if (mergesigma->presence == 1 && mergesigma->number > IDENTITY) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, mergesigma->number, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+
+ if ((fsm_state+i)->in != UNKNOWN && (fsm_state+i)->out == UNKNOWN) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma=start_mergesigma; mergesigma!=NULL; mergesigma=mergesigma->next) {
+ if ((mergesigma->presence == 1) && (mergesigma->number > IDENTITY)) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, mergesigma->number, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+
+ if ((fsm_state+i)->in == UNKNOWN && (fsm_state+i)->out == UNKNOWN) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ for (mergesigma2=start_mergesigma; mergesigma2 != NULL ; mergesigma2 = mergesigma2->next) {
+ for (mergesigma=start_mergesigma; mergesigma!=NULL; mergesigma=mergesigma->next) {
+ if (((mergesigma->presence == 1 && mergesigma2->presence == 1 && mergesigma->number > IDENTITY && mergesigma2->number > IDENTITY) || (mergesigma->number == UNKNOWN && mergesigma2->number > IDENTITY && mergesigma2->presence == 1) || (mergesigma2->number == UNKNOWN && mergesigma->number > IDENTITY && mergesigma->presence == 1)) && mergesigma->number != mergesigma2->number) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, mergesigma->number, mergesigma2->number, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+ }
+ }
+
+ /* Simply copy arcs that are not IDENTITY or UNKNOWN */
+ if (((fsm_state+i)->in > IDENTITY || (fsm_state+i)->in == EPSILON) && ((fsm_state+i)->out > IDENTITY || (fsm_state+i)->out == EPSILON)) {
+
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+
+ if ((fsm_state+i)->in == -1) {
+ add_fsm_arc(new_2_state, j, (fsm_state+i)->state_no, (fsm_state+i)->in, (fsm_state+i)->out, (fsm_state+i)->target, (fsm_state+i)->final_state, (fsm_state+i)->start_state);
+ j++;
+ }
+ }
+
+ add_fsm_arc(new_2_state, j, -1, -1, -1, -1, -1, -1);
+ xxfree(net2->states);
+ net2->states = new_2_state;
+ }
+ xxfree(mapping_1);
+ xxfree(mapping_2);
+
+ /* Free structure */
+ for (mergesigma2 = NULL; start_mergesigma != NULL; ) {
+ mergesigma2 = start_mergesigma;
+ start_mergesigma = start_mergesigma->next;
+ xxfree(mergesigma2);
+ }
+}
+
+
+int add_fsm_arc(struct fsm_state *fsm, int offset, int state_no, int in, int out, int target, int final_state, int start_state) {
+
+ (fsm+offset)->state_no = state_no;
+ (fsm+offset)->in = in;
+ (fsm+offset)->out= out;
+ (fsm+offset)->target = target;
+ (fsm+offset)->final_state = final_state;
+ (fsm+offset)->start_state = start_state;
+ offset++;
+ return(offset);
+}
+
+
+void fsm_count(struct fsm *net) {
+ struct fsm_state *fsm;
+ int i, linecount, arccount, oldstate, finalcount, maxstate;
+ linecount = arccount = finalcount = maxstate = 0;
+
+ oldstate = -1;
+
+ fsm = net->states;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->state_no > maxstate)
+ maxstate = (fsm+i)->state_no;
+
+ linecount++;
+ if ((fsm+i)->target != -1) {
+ arccount++;
+ // if (((fsm+i)->in != (fsm+i)->out) || ((fsm+i)->in == UNKNOWN) || ((fsm+i)->out == UNKNOWN))
+ // arity = 2;
+ }
+ if ((fsm+i)->state_no != oldstate) {
+ if ((fsm+i)->final_state) {
+ finalcount++;
+ }
+ oldstate = (fsm+i)->state_no;
+ }
+ }
+
+ linecount++;
+ net->statecount = maxstate+1;
+ net->linecount = linecount;
+ net->arccount = arccount;
+ net->finalcount = finalcount;
+}
+
+static void fsm_add_to_states(struct fsm *net, int add) {
+ struct fsm_state *fsm;
+ int i;
+
+ fsm=net->states;
+ for(i=0; (fsm+i)->state_no != -1; i++) {
+ (fsm+i)->state_no = (fsm+i)->state_no + add;
+ if ((fsm+i)->target != -1)
+ (fsm+i)->target = (fsm+i)->target + add;
+ }
+}
+
+struct fsm *fsm_concat_m_n(struct fsm *net1, int m, int n) {
+ struct fsm *acc;
+ int i;
+ acc = fsm_empty_string();
+ for (i = 1; i <= n ;i++) {
+ if (i > m)
+ acc = fsm_concat(acc, fsm_optionality(fsm_copy(net1)));
+ else
+ acc = fsm_concat(acc, fsm_copy(net1));
+ }
+ fsm_destroy(net1);
+ return(acc);
+}
+
+struct fsm *fsm_concat_n(struct fsm *net1, int n) {
+ return(fsm_concat_m_n(net1,n,n));
+}
+
+struct fsm *fsm_concat(struct fsm *net1, struct fsm *net2) {
+ struct fsm_state *fsm1, *fsm2, *new_fsm;
+ int i,j,current_final;
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm1 = net1->states;
+ fsm2 = net2->states;
+ fsm_count(net1);
+ fsm_count(net2);
+ /* The concatenation of a language with no final state should yield the empty language */
+ if ((net1->finalcount == 0) || (net2->finalcount == 0)) {
+ fsm_destroy(net1);
+ fsm_destroy(net2);
+ net1 = fsm_empty_set();
+ return(net1);
+ }
+
+ /* Add |fsm1| states to the state numbers of fsm2 */
+ fsm_add_to_states(net2, net1->statecount);
+
+ new_fsm = xxmalloc(((sizeof(struct fsm_state))*(net1->linecount + net2->linecount + net1->finalcount + 2 )));
+ current_final = -1;
+ /* Copy fsm1, fsm2 after each other, adding appropriate epsilon arcs */
+ for(i=0,j=0; (fsm1+i)->state_no != -1; i++) {
+ if (((fsm1+i)->final_state == 1) && ((fsm1+i)->state_no != current_final)) {
+ add_fsm_arc(new_fsm, j, (fsm1+i)->state_no, EPSILON, EPSILON, net1->statecount, 0, (fsm1+i)->start_state);
+ current_final = (fsm1+i)->state_no;
+ j++;
+ }
+ if (!(((fsm1+i)->target == -1) && ((fsm1+i)->final_state == 1))) {
+ add_fsm_arc(new_fsm, j, (fsm1+i)->state_no, (fsm1+i)->in, (fsm1+i)->out, (fsm1+i)->target, 0, (fsm1+i)->start_state);
+ j++;
+ }
+ }
+
+ for(i=0; (fsm2+i)->state_no != -1; i++, j++) {
+ add_fsm_arc(new_fsm, j, (fsm2+i)->state_no, (fsm2+i)->in, (fsm2+i)->out, (fsm2+i)->target, (fsm2+i)->final_state, 0);
+ }
+ add_fsm_arc(new_fsm, j, -1, -1, -1, -1, -1, -1);
+ xxfree(net1->states);
+ fsm_destroy(net2);
+ net1->states = new_fsm;
+ if (sigma_find_number(EPSILON, net1->sigma) == -1) {
+ sigma_add_special(EPSILON, net1->sigma);
+ }
+ fsm_count(net1);
+ net1->is_epsilon_free = NO;
+ net1->is_deterministic = NO;
+ net1->is_minimized = NO;
+ net1->is_pruned = NO;
+ return(fsm_minimize(net1));
+}
+
+struct fsm *fsm_union(struct fsm *net1, struct fsm *net2) {
+ struct fsm_state *new_fsm, *fsm1, *fsm2;
+ int i, j, net1_offset, net2_offset, new_target, arccount;
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm_count(net1);
+ fsm_count(net2);
+
+ fsm1 = net1->states;
+ fsm2 = net2->states;
+
+ net1_offset = 1;
+ net2_offset = net1->statecount + 1;
+ new_fsm = xxmalloc((net1->linecount + net2->linecount + 2) * sizeof(struct fsm_state));
+
+ j = 0;
+
+ add_fsm_arc(new_fsm, j++, 0, EPSILON, EPSILON, net1_offset, 0 , 1);
+ add_fsm_arc(new_fsm, j++, 0, EPSILON, EPSILON, net2_offset, 0 , 1);
+ arccount = 2;
+ for (i=0 ; (fsm1+i)->state_no != -1; i++) {
+ new_target = (fsm1+i)->target == -1 ? -1 : (fsm1+i)->target + net1_offset;
+ add_fsm_arc(new_fsm, j++, (fsm1+i)->state_no + net1_offset, (fsm1+i)->in, (fsm1+i)->out, new_target, (fsm1+i)->final_state, 0);
+ if (new_target != -1) arccount++;
+ }
+ for (i=0 ; (fsm2+i)->state_no != -1; i++) {
+ new_target = (fsm2+i)->target == -1 ? -1 : (fsm2+i)->target + net2_offset;
+ add_fsm_arc(new_fsm, j++, (fsm2+i)->state_no + net2_offset, (fsm2+i)->in, (fsm2+i)->out, new_target, (fsm2+i)->final_state, 0);
+ if (new_target != -1) arccount++;
+ }
+ add_fsm_arc(new_fsm, j++, -1, -1, -1, -1, -1, -1);
+ xxfree(net1->states);
+ net1->states = new_fsm;
+ net1->statecount = net1->statecount + net2->statecount + 1;
+ net1->linecount = j;
+ net1->arccount = arccount;
+ net1->finalcount = net1->finalcount + net2->finalcount;
+ fsm_destroy(net2);
+ fsm_update_flags(net1,NO,NO,NO,NO,UNK,NO);
+ if (sigma_find_number(EPSILON, net1->sigma) == -1) {
+ sigma_add_special(EPSILON, net1->sigma);
+ }
+ return(net1);
+}
+
+struct fsm *fsm_completes(struct fsm *net, int operation) {
+ struct fsm_state *fsm, *new_fsm;
+ int i, j, offset, statecount, sigsize, *state_table, sink_state, target, last_sigma = 0, arccount = 0, incomplete;
+ short *starts, *finals, *sinks;
+
+ /* TODO: this currently relies on that the sigma is gap-free in its numbering */
+ /* which can't always be counted on, especially when reading external machines */
+
+ /* TODO: check arity */
+
+ if (net->is_minimized != YES)
+ net = fsm_minimize(net);
+
+ incomplete = 0;
+ fsm = net->states;
+ if (sigma_find_number(UNKNOWN, net->sigma) != -1) {
+ sigma_remove("@_UNKNOWN_SYMBOL_@",net->sigma);
+ }
+ if (sigma_find_number(IDENTITY, net->sigma) == -1) {
+ sigma_add_special(IDENTITY, net->sigma);
+ incomplete = 1;
+ }
+
+ sigsize = sigma_size(net->sigma);
+ last_sigma = sigma_max(net->sigma);
+
+ if (sigma_find_number(EPSILON, net->sigma) != -1)
+ sigsize--;
+
+ fsm_count(net);
+ statecount = net->statecount;
+ starts = xxmalloc(sizeof(short)*(statecount+1)); /* +1 for sink state */
+ finals = xxmalloc(sizeof(short)*(statecount+1));
+ sinks = xxmalloc(sizeof(short)*(statecount+1));
+
+ /* Init starts, finals, sinks arrays */
+
+ for (i=0; i < statecount; i++) {
+ *(sinks+i) = 1;
+ *(finals+i) = 0;
+ *(starts+i) = 0;
+ }
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if (operation == COMPLEMENT) {
+ if ((fsm+i)->final_state == 1) {
+ (fsm+i)->final_state = 0;
+ } else if ((fsm+i)->final_state == 0) {
+ (fsm+i)->final_state = 1;
+ }
+ }
+ if ((fsm+i)->target != -1)
+ arccount++;
+ starts[(fsm+i)->state_no] = (fsm+i)->start_state;
+ finals[(fsm+i)->state_no] = (fsm+i)->final_state;
+ if ((fsm+i)->final_state && operation != COMPLEMENT)
+ *(sinks+((fsm+i)->state_no)) = 0;
+ if ((fsm+i)->final_state == 0 && (operation == COMPLEMENT))
+ *(sinks+((fsm+i)->state_no)) = 0;
+ if (((fsm+i)->target != -1) && ((fsm+i)->state_no != (fsm+i)->target))
+ *(sinks+((fsm+i)->state_no)) = 0;
+ }
+
+ net->is_loop_free = NO;
+ net->pathcount = PATHCOUNT_CYCLIC;
+
+ if (incomplete == 0 && (arccount == (sigsize)*statecount)) {
+ /* printf("Already complete!\n"); */
+
+/* if (operation == COMPLEMENT) { */
+/* for (i=0; (fsm+i)->state_no != -1; i++) { */
+/* if ((fsm+i)->final_state) { */
+/* (fsm+i)->final_state = 0; */
+/* } else { */
+/* (fsm+i)->final_state = 1; */
+/* } */
+/* } */
+/* } */
+ xxfree(starts);
+ xxfree(finals);
+ xxfree(sinks);
+ net->is_completed = YES;
+ net->is_minimized = YES;
+ net->is_pruned = NO;
+ net->is_deterministic = YES;
+ return(net);
+ }
+
+ /* Find an existing sink state, or invent a new one */
+
+ for (i=0, sink_state = -1; i<statecount; i++) {
+ if (sinks[i] == 1) {
+ sink_state = i;
+ break;
+ }
+ }
+
+ if (sink_state == -1) {
+ sink_state = statecount;
+ *(starts+sink_state) = 0;
+ if (operation == COMPLEMENT) {
+ *(finals+sink_state) = 1;
+ } else {
+ *(finals+sink_state) = 0;
+ }
+ statecount++;
+ }
+
+
+ /* We can build a state table without memory problems since the size */
+ /* of the completed machine will be |Sigma| * |States| in all cases */
+
+ sigsize += 2;
+
+ state_table = xxmalloc(sizeof(int)*sigsize*statecount);
+
+ /* Init state table */
+ /* i = state #, j = sigma # */
+ for (i=0; i<statecount; i++) {
+ for (j=0; j<sigsize; j++) {
+ *(state_table+(i*sigsize+j)) = -1;
+ }
+ }
+
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1) {
+ *(state_table+(((fsm+i)->state_no)*sigsize+((fsm+i)->in))) = (fsm+i)->target;
+ }
+ }
+ /* Add looping arcs from and to sink state */
+ for (j=2; j<=last_sigma; j++)
+ *(state_table+(sink_state*sigsize+j)) = sink_state;
+ /* Add missing arcs to sink state from all states */
+ for (i=0; i<statecount; i++) {
+ for (j=2; j<=last_sigma; j++) {
+ if (*(state_table+(i*sigsize+j)) == -1)
+ *(state_table+(i*sigsize+j)) = sink_state;
+ }
+ }
+
+ new_fsm = xxmalloc(sizeof(struct fsm_state)*(sigsize*statecount+1));
+
+/* Complement requires toggling final, nonfinal states */
+/* if (operation == COMPLEMENT) */
+/* for (i=0; i < statecount; i++) */
+/* *(finals+i) = *(finals+i) == 0 ? 1 : 0; */
+
+ for (i=0, offset = 0; i<statecount; i++) {
+ for (j=2; j<=last_sigma; j++) {
+ target = *(state_table+(i*sigsize+j)) == -1 ? sink_state : *(state_table+(i*sigsize+j));
+ add_fsm_arc(new_fsm, offset, i, j, j, target, finals[i], starts[i]);
+ offset++;
+ }
+ }
+ add_fsm_arc(new_fsm, offset, -1, -1, -1, -1, -1, -1);
+ offset++;
+ xxfree(net->states);
+ net->states = new_fsm;
+ xxfree(starts);
+ xxfree(finals);
+ xxfree(sinks);
+ xxfree(state_table);
+ net->is_minimized = NO;
+ net->is_pruned = NO;
+ net->is_completed = YES;
+ net->statecount = statecount;
+ return(net);
+}
+
+struct fsm *fsm_complete(struct fsm *net) {
+ return(fsm_completes(net, COMPLETE));
+}
+
+struct fsm *fsm_complement(struct fsm *net) {
+ return(fsm_completes(net, COMPLEMENT));
+}
+
+struct fsm *fsm_kleene_closure(struct fsm *net, int operation) {
+ struct fsm_state *fsm, *new_fsm;
+ int i, j, laststate, curr_state, curr_target, arccount;
+
+ if (operation == OPTIONALITY) {
+ return(fsm_union(net,fsm_empty_string()));
+ }
+
+ net = fsm_minimize(net);
+ fsm_count(net);
+
+ fsm = net->states;
+
+ new_fsm = xxmalloc( (net->linecount + net->finalcount + 1) * sizeof(struct fsm_state));
+
+ j = 0;
+ if (operation == KLEENE_STAR)
+ add_fsm_arc(new_fsm, j++, 0, EPSILON, EPSILON, 1, 1, 1);
+ if (operation == KLEENE_PLUS)
+ add_fsm_arc(new_fsm, j++, 0, EPSILON, EPSILON, 1, 0, 1);
+ laststate = 0;
+ arccount = 1;
+ for (i = 0 ; (fsm+i)->state_no != -1; i++, laststate = curr_state) {
+ curr_state = (fsm+i)->state_no + 1;
+ curr_target = (fsm+i)->target == -1 ? -1 : (fsm+i)->target + 1;
+ if (curr_target == -1 && (fsm+i)->final_state == 1) {
+ add_fsm_arc(new_fsm, j++, curr_state, EPSILON, EPSILON, 0, 1, 0);
+ arccount++;
+ continue;
+ }
+ if (curr_state != laststate && (fsm+i)->final_state == 1) {
+ arccount++;
+ add_fsm_arc(new_fsm, j++, curr_state, EPSILON, EPSILON, 0, 1, 0);
+ }
+ add_fsm_arc(new_fsm, j++, curr_state, (fsm+i)->in, (fsm+i)->out, curr_target, (fsm+i)->final_state, 0);
+ if (curr_target != -1) arccount++;
+ }
+ add_fsm_arc(new_fsm, j++, -1,-1,-1,-1,-1,-1);
+ net->statecount = net->statecount+1;
+ net->linecount = j;
+ net->finalcount = operation == KLEENE_STAR ? net->finalcount+1 : net->finalcount;
+ net->arccount = arccount;
+ net->pathcount = PATHCOUNT_UNKNOWN;
+ xxfree(net->states);
+ net->states = new_fsm;
+ if (sigma_find_number(EPSILON, net->sigma) == -1)
+ sigma_add_special(EPSILON, net->sigma);
+ fsm_update_flags(net,NO,NO,NO,NO,UNK,NO);
+ return(net);
+}
+
+char *fsm_network_to_char(struct fsm *net) {
+ struct sigma *sigma, *sigprev;
+ sigma = net->sigma;
+ if (sigma->number == -1) {
+ return NULL;
+ }
+ for (; sigma != NULL && sigma->number != -1 ; sigma = sigma->next) {
+ sigprev = sigma;
+ }
+ return(strdup(sigprev->symbol));
+}
+
+struct fsm *fsm_substitute_label(struct fsm *net, char *original, struct fsm *substitute) {
+
+ struct fsm *outnet, *subnet2;
+ struct fsm_read_handle *inh, *subh, *subh2;
+ struct fsm_construct_handle *outh;
+ char *subin, *subout;
+ int i, repsym, source, target, in, out, addstate1, addstate2;
+
+ fsm_merge_sigma(net, substitute);
+ addstate1 = net->statecount;
+ addstate2 = substitute->statecount;
+
+ inh = fsm_read_init(net);
+ subh = fsm_read_init(substitute);
+ repsym = fsm_get_symbol_number(inh, original);
+ if (repsym == -1) {
+ fsm_read_done(inh);
+ return(net);
+ }
+ outh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(outh, net->sigma);
+ while (fsm_get_next_arc(inh)) {
+ source = fsm_get_arc_source(inh);
+ target = fsm_get_arc_target(inh);
+ in = fsm_get_arc_num_in(inh);
+ out = fsm_get_arc_num_out(inh);
+
+ /* Double-sided arc, splice in substitute network */
+ if (in == repsym && out == repsym) {
+ fsm_read_reset(subh);
+ fsm_construct_add_arc_nums(outh, source, addstate1, EPSILON, EPSILON);
+ while (fsm_get_next_arc(subh)) {
+ source = fsm_get_arc_source(subh);
+ target = fsm_get_arc_target(subh);
+ subin = fsm_get_arc_in(subh);
+ subout = fsm_get_arc_out(subh);
+ fsm_construct_add_arc(outh, source+addstate1, target+addstate1, subin, subout);
+ }
+ while ((i = fsm_get_next_final(subh)) != -1) {
+ target = fsm_get_arc_target(inh);
+ fsm_construct_add_arc_nums(outh, addstate1+i, target, EPSILON, EPSILON);
+ }
+ addstate1 = addstate1 + addstate2;
+ /* One-sided replace, splice in repsym .x. sub or sub .x. repsym */
+ } else if (in == repsym || out == repsym) {
+ if (in == repsym) {
+ subnet2 = fsm_minimize(fsm_cross_product(fsm_copy(substitute), fsm_symbol(fsm_get_arc_out(inh))));
+ } else {
+ subnet2 = fsm_minimize(fsm_cross_product(fsm_symbol(fsm_get_arc_in(inh)),fsm_copy(substitute)));
+ }
+ fsm_construct_add_arc_nums(outh, source, addstate1, EPSILON, EPSILON);
+ subh2 = fsm_read_init(subnet2);
+ while (fsm_get_next_arc(subh2)) {
+ source = fsm_get_arc_source(subh2);
+ target = fsm_get_arc_target(subh2);
+ subin = fsm_get_arc_in(subh2);
+ subout = fsm_get_arc_out(subh2);
+ fsm_construct_add_arc(outh, source+addstate1, target+addstate1, subin, subout);
+ }
+ while ((i = fsm_get_next_final(subh2)) != -1) {
+ target = fsm_get_arc_target(inh);
+ fsm_construct_add_arc_nums(outh, addstate1+i, target, EPSILON, EPSILON);
+ }
+ fsm_read_done(subh2);
+ addstate1 = addstate1 + subnet2->statecount;
+ fsm_destroy(subnet2);
+ } else {
+ /* Default, just copy arc */
+ fsm_construct_add_arc_nums(outh, source, target, in, out);
+ }
+ }
+
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(outh, i);
+ }
+ while ((i = fsm_get_next_initial(inh)) != -1) {
+ fsm_construct_set_initial(outh, i);
+ }
+ fsm_read_done(inh);
+ fsm_read_done(subh);
+ outnet = fsm_construct_done(outh);
+ return(outnet);
+}
+
+struct fsm *fsm_substitute_symbol(struct fsm *net, char *original, char *substitute) {
+ struct fsm_state *fsm;
+ int i,o,s = EPSILON;
+ if (strcmp(original,substitute) == 0)
+ return(net);
+ if ((o = sigma_find(original, net->sigma)) == -1) {
+ //fprintf(stderr, "\nSymbol '%s' not found in network!\n", original);
+ return(net);
+ }
+ if (strcmp(substitute,"0") == 0)
+ s = EPSILON;
+ else if (substitute != NULL && (s = sigma_find(substitute, net->sigma)) == -1) {
+ s = sigma_add(substitute, net->sigma);
+ }
+ for (i=0, fsm = net->states; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->in == o) {
+ (fsm+i)->in = s;
+ }
+ if ((fsm+i)->out == o) {
+ (fsm+i)->out = s;
+ }
+ }
+ net->sigma = sigma_remove(original, net->sigma);
+ sigma_sort(net);
+ fsm_update_flags(net, NO, NO, NO, NO, NO, NO);
+ sigma_cleanup(net,0);
+ /* if s = epsilon */
+ net->is_minimized = NO;
+ return(fsm_determinize(net));
+}
+
+struct fsm *fsm_cross_product(struct fsm *net1, struct fsm *net2) {
+ int i, a, b, current_state, current_start, current_final, target_number, symbol1, symbol2, epsilon = 0, unknown = 0;
+ struct fsm_state *machine_a, *machine_b, *fsm;
+ struct state_arr *point_a, *point_b;
+ struct triplethash *th;
+
+ /* Perform a cross product by running two machines in parallel */
+ /* The approach here allows a state to stay, creating a a:0 or 0:b transition */
+ /* with the a/b-state waiting, and the arc going to {a,stay} or {stay,b} */
+ /* the wait maneuver is only possible if the waiting state is final */
+
+ /* For the rewrite rules compilation, a different cross-product is used: */
+ /* rewrite_cp() synchronizes A and B as long as possible to get a unique */
+ /* output match for each cross product. */
+
+ /* This behavior where we postpone zeroes on either side and perform */
+ /* and equal length cross-product as long as possible and never intermix */
+ /* ?:0 and 0:? arcs (i.e. we keep both machines synchronized as long as possible */
+ /* can be done by [A .x. B] & ?:?* [?:0*|0:?*] at the cost of possibly */
+ /* up to three times larger transducers. */
+ /* This is very similar to the idea in "tristate composition" in fsm_compose() */
+
+ /* This function is only used for explicit cross products */
+ /* such as a:b or A.x.B, etc. In rewrite rules, we use rewrite_cp() */
+
+ net1 = fsm_minimize(net1);
+ net2 = fsm_minimize(net2);
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm_count(net1);
+ fsm_count(net2);
+
+ machine_a = net1->states;
+ machine_b = net2->states;
+
+ /* new state 0 = {0,0} */
+
+ STACK_2_PUSH(0,0);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 0, 0, 0);
+
+ fsm_state_init(sigma_max(net1->sigma));
+
+ point_a = init_state_pointers(machine_a);
+ point_b = init_state_pointers(machine_b);
+
+ while (!int_stack_isempty()) {
+
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ b = int_stack_pop();
+
+ /* printf("Treating pair: {%i,%i}\n",a,b); */
+
+ current_state = triplet_hash_find(th, a, b, 0);
+ current_start = (((point_a+a)->start == 1) && ((point_b+b)->start == 1)) ? 1 : 0;
+ current_final = (((point_a+a)->final == 1) && ((point_b+b)->final == 1)) ? 1 : 0;
+
+ fsm_state_set_current_state(current_state, current_final, current_start);
+
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ for (machine_b = (point_b+b)->transitions; machine_b->state_no == b ; machine_b++) {
+
+ if ((machine_a->target == -1) && (machine_b->target == -1)) {
+ continue;
+ }
+ if ((machine_a->target == -1) && (machine_a->final_state == 0)) {
+ continue;
+ }
+ if ((machine_b->target == -1) && (machine_b->final_state == 0)) {
+ continue;
+ }
+ /* Main check */
+ if (!((machine_a->target == -1) || (machine_b->target == -1))) {
+ if ((target_number = triplet_hash_find(th, machine_a->target, machine_b->target, 0)) == -1) {
+ STACK_2_PUSH(machine_b->target, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, machine_b->target, 0);
+ }
+ symbol1 = machine_a->in;
+ symbol2 = machine_b->in;
+ if (symbol1 == IDENTITY && symbol2 != IDENTITY)
+ symbol1 = UNKNOWN;
+ if (symbol2 == IDENTITY && symbol1 != IDENTITY)
+ symbol2 = UNKNOWN;
+
+ fsm_state_add_arc(current_state, symbol1, symbol2, target_number, current_final, current_start);
+ /* @:@ -> @:@ and also ?:? */
+ if ((machine_a->in == IDENTITY) && (machine_b->in == IDENTITY)) {
+ fsm_state_add_arc(current_state, UNKNOWN, UNKNOWN, target_number, current_final, current_start);
+ }
+ }
+ if (machine_a->final_state == 1 && machine_b->target != -1) {
+
+ /* Add 0:b i.e. stay in state A */
+ if ((target_number = triplet_hash_find(th, machine_a->state_no, machine_b->target, 0)) == -1) {
+ STACK_2_PUSH(machine_b->target, machine_a->state_no);
+ target_number = triplet_hash_insert(th, machine_a->state_no, machine_b->target, 0);
+ }
+ /* @:0 becomes ?:0 */
+ symbol2 = machine_b->in == IDENTITY ? UNKNOWN : machine_b->in;
+ fsm_state_add_arc(current_state, EPSILON, symbol2, target_number, current_final, current_start);
+ }
+
+ if (machine_b->final_state == 1 && machine_a->target != -1) {
+
+ /* Add a:0 i.e. stay in state B */
+ if ((target_number = triplet_hash_find(th, machine_a->target, machine_b->state_no, 0)) == -1) {
+ STACK_2_PUSH(machine_b->state_no, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, machine_b->state_no, 0);
+ }
+ /* @:0 becomes ?:0 */
+ symbol1 = machine_a->in == IDENTITY ? UNKNOWN : machine_a->in;
+ fsm_state_add_arc(current_state, symbol1, EPSILON, target_number, current_final, current_start);
+ }
+ }
+ }
+ /* Check arctrack */
+ fsm_state_end_state();
+ }
+
+ xxfree(net1->states);
+ fsm_state_close(net1);
+
+ for (i=0, fsm = net1->states; (fsm+i)->state_no != -1; i++) {
+ if (((fsm+i)->in == EPSILON) || ((fsm+i)->out == EPSILON))
+ epsilon = 1;
+ if (((fsm+i)->in == UNKNOWN) || ((fsm+i)->out == UNKNOWN))
+ unknown = 1;
+ }
+ if (epsilon == 1) {
+ if (sigma_find_number(EPSILON, net1->sigma) == -1) {
+ sigma_add_special(EPSILON, net1->sigma);
+ }
+ }
+ if (unknown == 1) {
+ if (sigma_find_number(UNKNOWN, net1->sigma) == -1) {
+ sigma_add_special(UNKNOWN, net1->sigma);
+ }
+ }
+ xxfree(point_a);
+ xxfree(point_b);
+ fsm_destroy(net2);
+ triplet_hash_free(th);
+ return(fsm_coaccessible(net1));
+}
+
+struct fsm *fsm_precedes(struct fsm *net1, struct fsm *net2) {
+ return(fsm_complement(fsm_minimize(fsm_contains(fsm_minimize(fsm_concat(fsm_minimize(fsm_copy(net2)),fsm_concat(fsm_universal(),fsm_minimize(fsm_copy(net1)))))))));
+}
+
+struct fsm *fsm_follows(struct fsm *net1, struct fsm *net2) {
+ return(fsm_complement(fsm_minimize(fsm_contains(fsm_minimize(fsm_concat(fsm_minimize(fsm_copy(net1)),fsm_concat(fsm_universal(),fsm_minimize(fsm_copy(net2)))))))));
+}
+
+struct fsm *fsm_unflatten(struct fsm *net, char *epsilon_sym, char *repeat_sym) {
+ int a, b, current_state, current_start, current_final, target_number, epsilon, repeat, in, out;
+ struct fsm_state *even_state, *odd_state;
+ struct state_arr *point_a;
+ struct triplethash *th;
+
+ fsm_minimize(net);
+ fsm_count(net);
+
+ epsilon = sigma_find(epsilon_sym, net->sigma);
+ repeat = sigma_find(repeat_sym, net->sigma);
+
+ even_state = net->states;
+
+ /* new state 0 = {0,0} */
+
+ STACK_2_PUSH(0,0);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 0, 0, 0);
+
+ fsm_state_init(sigma_max(net->sigma));
+
+ point_a = init_state_pointers(even_state);
+
+ while (!int_stack_isempty()) {
+
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ a = int_stack_pop();
+
+ /* printf("Treating pair: {%i,%i}\n",a,b); */
+
+ current_state = triplet_hash_find(th, a, a, 0);
+ current_start = ((point_a+a)->start == 1) ? 1 : 0;
+ current_final = ((point_a+a)->final == 1) ? 1 : 0;
+
+ fsm_state_set_current_state(current_state, current_final, current_start);
+
+ for (even_state = (point_a+a)->transitions; even_state->state_no == a; even_state++) {
+ if (even_state->target == -1) {
+ continue;
+ }
+ in = even_state->in;
+ b = even_state->target;
+ for (odd_state = (point_a+b)->transitions; odd_state->state_no == b; odd_state++) {
+ if (odd_state->target == -1) {
+ continue;
+ }
+ if ((target_number = triplet_hash_find(th, odd_state->target, odd_state->target, 0)) == -1) {
+ STACK_2_PUSH(odd_state->target, odd_state->target);
+ target_number = triplet_hash_insert(th, odd_state->target, odd_state->target, 0);
+ }
+ in = even_state->in;
+ out = odd_state->in;
+ if (out == repeat) {
+ out = in;
+ } else if (in == IDENTITY || out == IDENTITY) {
+ in = in == IDENTITY ? UNKNOWN : in;
+ out = out == IDENTITY ? UNKNOWN : out;
+ }
+ if (in == epsilon) {
+ in = EPSILON;
+ }
+ if (out == epsilon) {
+ out = EPSILON;
+ }
+ fsm_state_add_arc(current_state, in, out, target_number, current_final, current_start);
+ }
+ }
+ fsm_state_end_state();
+ }
+ xxfree(net->states);
+ fsm_state_close(net);
+ xxfree(point_a);
+ triplet_hash_free(th);
+ return(net);
+}
+
+
+struct fsm *fsm_shuffle(struct fsm *net1, struct fsm *net2) {
+ int a, b, current_state, current_start, current_final, target_number;
+ struct fsm_state *machine_a, *machine_b;
+ struct state_arr *point_a, *point_b;
+ struct triplethash *th;
+
+ /* Shuffle A and B by making alternatively A move and B stay at each or */
+ /* vice versa at each step */
+
+ fsm_minimize(net1);
+ fsm_minimize(net2);
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm_count(net1);
+ fsm_count(net2);
+
+ machine_a = net1->states;
+ machine_b = net2->states;
+
+ /* new state 0 = {0,0} */
+
+ STACK_2_PUSH(0,0);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 0, 0, 0);
+
+ fsm_state_init(sigma_max(net1->sigma));
+
+ point_a = init_state_pointers(machine_a);
+ point_b = init_state_pointers(machine_b);
+
+ while (!int_stack_isempty()) {
+
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ b = int_stack_pop();
+
+ /* printf("Treating pair: {%i,%i}\n",a,b); */
+
+ current_state = triplet_hash_find(th, a, b, 0);
+ current_start = (((point_a+a)->start == 1) && ((point_b+b)->start == 1)) ? 1 : 0;
+ current_final = (((point_a+a)->final == 1) && ((point_b+b)->final == 1)) ? 1 : 0;
+
+ fsm_state_set_current_state(current_state, current_final, current_start);
+
+ /* Follow A, B stays */
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ if (machine_a->target == -1) {
+ continue;
+ }
+ if ((target_number = triplet_hash_find(th, machine_a->target, b, 0)) == -1) {
+ STACK_2_PUSH(b, machine_a->target);
+ target_number = triplet_hash_insert(th, machine_a->target, b, 0);
+ }
+
+ fsm_state_add_arc(current_state, machine_a->in, machine_a->out, target_number, current_final, current_start);
+ }
+
+ /* Follow B, A stays */
+ for (machine_b = (point_b+b)->transitions; machine_b->state_no == b ; machine_b++) {
+
+ if (machine_b->target == -1) {
+ continue;
+ }
+
+ if ((target_number = triplet_hash_find(th, a, machine_b->target, 0)) == -1) {
+ STACK_2_PUSH(machine_b->target, a);
+ target_number = triplet_hash_insert(th, a, machine_b->target, 0);
+ }
+ fsm_state_add_arc(current_state, machine_b->in, machine_b->out, target_number, current_final, current_start);
+ }
+
+ /* Check arctrack */
+ fsm_state_end_state();
+ }
+
+ xxfree(net1->states);
+ fsm_state_close(net1);
+ xxfree(point_a);
+ xxfree(point_b);
+ fsm_destroy(net2);
+ triplet_hash_free(th);
+ return(net1);
+}
+
+int fsm_equivalent(struct fsm *net1, struct fsm *net2) {
+ /* Test path equivalence of two FSMs by traversing both in parallel */
+ int a, b, matching_arc, equivalent;
+ struct fsm_state *machine_a, *machine_b;
+ struct state_arr *point_a, *point_b;
+ struct triplethash *th;
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm_count(net1);
+ fsm_count(net2);
+
+ machine_a = net1->states;
+ machine_b = net2->states;
+
+ equivalent = 0;
+ /* new state 0 = {0,0} */
+ STACK_2_PUSH(0,0);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 0, 0, 0);
+
+ point_a = init_state_pointers(machine_a);
+ point_b = init_state_pointers(machine_b);
+
+ while (!int_stack_isempty()) {
+
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ b = int_stack_pop();
+
+ if ((point_a+a)->final != (point_b+b)->final) {
+ goto not_equivalent;
+ }
+ /* Check that all arcs in A have matching arc in B, push new state pair on stack */
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ if (machine_a->target == -1) {
+ break;
+ }
+ matching_arc = 0;
+ for (machine_b = (point_b+b)->transitions; machine_b->state_no == b ; machine_b++) {
+ if (machine_b->target == -1) {
+ break;
+ }
+ if (machine_a->in == machine_b->in && machine_a->out == machine_b->out) {
+ matching_arc = 1;
+ if ((triplet_hash_find(th, machine_a->target, machine_b->target, 0)) == -1) {
+ STACK_2_PUSH(machine_b->target, machine_a->target);
+ triplet_hash_insert(th, machine_a->target, machine_b->target, 0);
+ }
+ break;
+ }
+ }
+ if (matching_arc == 0) {
+ goto not_equivalent;
+ }
+ }
+ for (machine_b = (point_b+b)->transitions; machine_b->state_no == b ; machine_b++) {
+ if (machine_b->target == -1) {
+ break;
+ }
+ matching_arc = 0;
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ if (machine_a->in == machine_b->in && machine_a->out == machine_b->out) {
+ matching_arc = 1;
+ break;
+ }
+ }
+ if (matching_arc == 0) {
+ goto not_equivalent;
+ }
+ }
+ }
+ equivalent = 1;
+ not_equivalent:
+ fsm_destroy(net1);
+ fsm_destroy(net2);
+ xxfree(point_a);
+ xxfree(point_b);
+ triplet_hash_free(th);
+ return(equivalent);
+}
+
+
+struct fsm *fsm_minus(struct fsm *net1, struct fsm *net2) {
+ int a, b, current_state, current_start, current_final, target_number, b_has_trans, btarget, statecount;
+ struct fsm_state *machine_a, *machine_b;
+ struct state_arr *point_a, *point_b;
+ struct triplethash *th;
+ statecount = 0;
+
+ net1 = fsm_minimize(net1);
+ net2 = fsm_minimize(net2);
+
+ fsm_merge_sigma(net1, net2);
+
+ fsm_count(net1);
+ fsm_count(net2);
+
+ machine_a = net1->states;
+ machine_b = net2->states;
+
+ /* new state 0 = {1,1} */
+
+ int_stack_clear();
+ STACK_2_PUSH(1,1);
+
+ th = triplet_hash_init();
+ triplet_hash_insert(th, 1, 1, 0);
+
+ point_a = init_state_pointers(machine_a);
+ point_b = init_state_pointers(machine_b);
+
+ fsm_state_init(sigma_max(net1->sigma));
+
+ while (!int_stack_isempty()) {
+ statecount++;
+ /* Get a pair of states to examine */
+
+ a = int_stack_pop();
+ b = int_stack_pop();
+
+ current_state = triplet_hash_find(th, a, b, 0);
+ a--;
+ b--;
+
+ if (b == -1) {
+ current_start = 0;
+ current_final = (point_a+a)->final;
+ } else {
+ current_start = (a == 0 && b == 0) ? 1 : 0;
+ current_final = (((point_a+a)->final == 1) && ((point_b+b)->final == 0)) ? 1 : 0;
+ }
+
+ fsm_state_set_current_state(current_state, current_final, current_start);
+
+ for (machine_a = (point_a+a)->transitions ; machine_a->state_no == a ; machine_a++) {
+ if (machine_a->target == -1) {
+ break;
+ continue;
+ }
+ if (b == -1) {
+ /* b is dead */
+ if ((target_number = triplet_hash_find(th, (machine_a->target)+1, 0, 0)) == -1) {
+ STACK_2_PUSH(0, (machine_a->target)+1);
+ target_number = triplet_hash_insert(th, (machine_a->target)+1, 0, 0);
+ }
+ } else {
+ /* b is alive */
+ b_has_trans = 0;
+ for (machine_b = (point_b+b)->transitions ; machine_b->state_no == b ; machine_b++) {
+ if (machine_a->in == machine_b->in && machine_a->out == machine_b->out) {
+ b_has_trans = 1;
+ btarget = machine_b->target;
+ break;
+ }
+ }
+ if (b_has_trans) {
+ if ((target_number = triplet_hash_find(th, (machine_a->target)+1, btarget+1, 0)) == -1) {
+ STACK_2_PUSH(btarget+1, (machine_a->target)+1);
+ target_number = triplet_hash_insert(th, (machine_a->target)+1, (machine_b->target)+1, 0);
+ }
+ } else {
+ /* b is dead */
+ if ((target_number = triplet_hash_find(th, (machine_a->target)+1, 0, 0)) == -1) {
+ STACK_2_PUSH(0, (machine_a->target)+1);
+ target_number = triplet_hash_insert(th, (machine_a->target)+1, 0, 0);
+ }
+ }
+ }
+ fsm_state_add_arc(current_state, machine_a->in, machine_a->out, target_number, current_final, current_start);
+ }
+ fsm_state_end_state();
+ }
+
+ xxfree(net1->states);
+ fsm_state_close(net1);
+ xxfree(point_a);
+ xxfree(point_b);
+ fsm_destroy(net2);
+ triplet_hash_free(th);
+ return(fsm_minimize(net1));
+}
+
+struct fsm *fsm_contains(struct fsm *net) {
+ /* [?* A ?*] */
+ struct fsm *net2;
+
+ net2 = fsm_concat(fsm_concat(fsm_universal(),net),fsm_universal());
+ return(net2);
+}
+
+struct fsm *fsm_universal() {
+ struct fsm *net;
+ int s;
+ net = fsm_create("");
+ fsm_update_flags(net, YES, YES, YES, YES, NO, NO);
+ net->states = xxmalloc(sizeof(struct fsm_state)*2);
+ s = sigma_add_special(IDENTITY,net->sigma);
+ add_fsm_arc(net->states, 0, 0, s, s, 0, 1, 1);
+ add_fsm_arc(net->states, 1, -1, -1, -1, -1, -1, -1);
+ net->arccount = 1;
+ net->statecount = 1;
+ net->linecount = 2;
+ net->finalcount = 1;
+ net->pathcount = PATHCOUNT_CYCLIC;
+ return(net);
+}
+
+struct fsm *fsm_contains_one(struct fsm *net) {
+ /* $A - $[[?+ A ?* & A ?*] | [A ?+ & A]] */
+ struct fsm *ret;
+ ret = fsm_minus(fsm_contains(fsm_copy(net)),fsm_contains(fsm_union(fsm_intersect(fsm_concat(fsm_kleene_plus(fsm_identity()),fsm_concat(fsm_copy(net),fsm_universal())) , fsm_concat(fsm_copy(net),fsm_universal())),fsm_intersect(fsm_concat(fsm_copy(net),fsm_kleene_plus(fsm_identity())), fsm_copy(net)))));
+ fsm_destroy(net);
+ return(ret);
+}
+
+struct fsm *fsm_contains_opt_one(struct fsm *net) {
+ /* $.A | ~$A */
+ struct fsm *ret;
+ ret = fsm_union(fsm_contains_one(fsm_copy(net)),fsm_complement(fsm_contains(fsm_copy(net))));
+ fsm_destroy(net);
+ return(ret);
+}
+
+struct fsm *fsm_simple_replace(struct fsm *net1, struct fsm *net2) {
+ /* [~[?* [A-0] ?*] [A.x.B]]* ~[?* [A-0] ?*] */
+
+ struct fsm *UPlus, *ret;
+ UPlus = fsm_minimize(fsm_kleene_plus(fsm_identity()));
+ ret = fsm_concat(fsm_minimize(fsm_kleene_star(fsm_minimize(fsm_concat(fsm_complement(fsm_minimize(fsm_concat(fsm_concat(fsm_universal(),fsm_minimize(fsm_intersect(fsm_copy(net1),fsm_copy(UPlus)))),fsm_universal()))),fsm_minimize(fsm_cross_product(fsm_copy(net1),fsm_copy(net2))))))),fsm_minimize(fsm_complement(fsm_minimize(fsm_concat(fsm_concat(fsm_universal(), fsm_intersect(fsm_copy(net1),fsm_copy(UPlus))),fsm_universal())))));
+ fsm_destroy(net1);
+ fsm_destroy(net2);
+ fsm_destroy(UPlus);
+ return(ret);
+}
+
+struct fsm *fsm_priority_union_upper(struct fsm *net1, struct fsm *net2) {
+ /* A .P. B = A | [~[A.u] .o. B] */
+ struct fsm *ret;
+ ret = fsm_union(fsm_copy(net1),fsm_compose(fsm_complement(fsm_upper(fsm_copy(net1))),net2));
+ fsm_destroy(net1);
+ return(ret);
+}
+
+struct fsm *fsm_priority_union_lower(struct fsm *net1, struct fsm *net2) {
+ /* A .p. B = A | B .o. ~[A.l] */
+ struct fsm *ret;
+ ret = fsm_union(fsm_copy(net1),fsm_compose(net2,fsm_complement(fsm_lower(fsm_copy(net1)))));
+ fsm_destroy(net1);
+ return(ret);
+}
+
+struct fsm *fsm_lenient_compose(struct fsm *net1, struct fsm *net2) {
+ /* A .O. B = [A .o. B] .P. B */
+ struct fsm *ret;
+ ret = fsm_priority_union_upper(fsm_compose(fsm_copy(net1),net2),fsm_copy(net1));
+ fsm_destroy(net1);
+ return(ret);
+}
+
+struct fsm *fsm_term_negation(struct fsm *net1) {
+ return(fsm_intersect(fsm_identity(),fsm_complement(net1)));
+}
+
+struct fsm *fsm_quotient_interleave(struct fsm *net1, struct fsm *net2) {
+ /* A/\/B = The set of strings you can interleave in B and get a string from A */
+ /* [B/[x \x* x] & A/x .o. [[[\x]:0]* (x:0 \x* x:0)]*].l */
+ struct fsm *Result;
+ Result = fsm_lower(fsm_compose(fsm_intersect(fsm_ignore(net2,fsm_concat(fsm_symbol("@>@"),fsm_concat(fsm_kleene_star(fsm_term_negation(fsm_symbol("@>@"))),fsm_symbol("@>@"))),OP_IGNORE_ALL),fsm_ignore(net1,fsm_symbol("@>@"),OP_IGNORE_ALL)),fsm_kleene_star(fsm_concat(fsm_kleene_star(fsm_cross_product(fsm_term_negation(fsm_symbol("@>@")),fsm_empty_string())),fsm_optionality(fsm_concat(fsm_cross_product(fsm_symbol("@>@"),fsm_empty_string()),fsm_concat(fsm_kleene_star(fsm_term_negation(f [...]
+
+ Result->sigma = sigma_remove("@>@",Result->sigma);
+ /* Could clean up sigma */
+ return(Result);
+}
+
+struct fsm *fsm_quotient_left(struct fsm *net1, struct fsm *net2) {
+ /* A\\\B = [B .o. A:0 ?*].l; */
+ /* A\\\B = the set of suffixes you can add to A to get a string in B */
+ struct fsm *Result;
+ Result = fsm_lower(fsm_compose(net2,fsm_concat(fsm_cross_product(net1,fsm_empty_string()),fsm_universal())));
+ return(Result);
+}
+
+struct fsm *fsm_quotient_right(struct fsm *net1, struct fsm *net2) {
+ struct fsm *Result;
+
+ /* A///B = [A .o. ?* B:0].l; */
+ /* A///B = the set of prefixes you can add to B to get strings in A */
+ Result = fsm_lower(fsm_compose(net1, fsm_concat(fsm_universal(),fsm_cross_product(net2,fsm_empty_string()))));
+ return(Result);
+}
+
+struct fsm *fsm_ignore(struct fsm *net1, struct fsm *net2, int operation) {
+ struct fsm_state *fsm1, *fsm2, *new_fsm;
+ struct fsm *Result;
+ short *handled_states1, *handled_states2;
+ int i, j, k, state_add_counter = 0, malloc_size, splices = 0, returns, target, splice_size, start_splice, states1, states2, lines1, lines2, *return_state;
+
+ net1 = fsm_minimize(net1);
+ net2 = fsm_minimize(net2);
+
+ if (fsm_isempty(net2)) {
+ fsm_destroy(net2);
+ return(net1);
+ }
+ fsm_merge_sigma(net1, net2);
+
+ fsm_count(net1);
+ fsm_count(net2);
+
+ states1 = net1->statecount;
+ states2 = net2->statecount;
+ lines1 = net1->linecount;
+ lines2 = net2->linecount;
+ fsm1 = net1->states;
+ fsm2 = net2->states;
+
+ if (operation == OP_IGNORE_INTERNAL) {
+ Result = fsm_lower(fsm_compose(fsm_ignore(fsm_copy(net1),fsm_symbol("@i<@"),OP_IGNORE_ALL),fsm_compose(fsm_complement(fsm_union(fsm_concat(fsm_symbol("@i<@"),fsm_universal()),fsm_concat(fsm_universal(),fsm_symbol("@i<@")))),fsm_simple_replace(fsm_symbol("@i<@"),fsm_copy(net2)))));
+ Result->sigma = sigma_remove("@i<@",Result->sigma);
+ fsm_destroy(net1);
+ fsm_destroy(net2);
+ return(Result);
+ }
+
+ malloc_size = lines1 + (states1 * (lines2 + net2->finalcount + 1));
+ new_fsm = xxmalloc(sizeof(struct fsm_state)*(malloc_size+1));
+
+ /* Mark if a state has been handled with ignore */
+ handled_states1 = xxmalloc(sizeof(short)*states1);
+ handled_states2 = xxmalloc(sizeof(short)*states2);
+
+ /* Mark which ignores return to which state */
+ return_state = xxmalloc(sizeof(int)*states1);
+ splice_size = states2;
+ start_splice = states1;
+ for (k=0; k<states1; k++)
+ *(handled_states1+k) = 0;
+
+ for (i=0, j=0; (fsm1+i)->state_no != -1; i++) {
+ if (*(handled_states1+(fsm1+i)->state_no) == 0) {
+ target = start_splice + splices * splice_size;
+ add_fsm_arc(new_fsm, j, (fsm1+i)->state_no, EPSILON, EPSILON, target, (fsm1+i)->final_state, (fsm1+i)->start_state);
+ *(return_state+splices) = (fsm1+i)->state_no;
+ *(handled_states1+(fsm1+i)->state_no) = 1;
+ j++;
+ splices++;
+ if ((fsm1+i)->in != -1) {
+ add_fsm_arc(new_fsm, j, (fsm1+i)->state_no, (fsm1+i)->in, (fsm1+i)->out, (fsm1+i)->target, (fsm1+i)->final_state, (fsm1+i)->start_state);
+ j++;
+ }
+ } else {
+ add_fsm_arc(new_fsm, j, (fsm1+i)->state_no, (fsm1+i)->in, (fsm1+i)->out, (fsm1+i)->target, (fsm1+i)->final_state, (fsm1+i)->start_state);
+ j++;
+ }
+ }
+
+ /* Add a sequence of fsm2s at the end, with arcs back to the appropriate states */
+
+ state_add_counter = start_splice;
+
+ for (returns = 0; splices>0; splices--, returns++) {
+ /* Zero handled return arc states */
+
+ for (k=0; k<states2; k++)
+ *(handled_states2+k) = 0;
+
+ for (i=0; (fsm2+i)->state_no != -1; i++) {
+ if ((fsm2+i)->final_state == 1 && *(handled_states2+(fsm2+i)->state_no) == 0) {
+ add_fsm_arc(new_fsm, j, (fsm2+i)->state_no + state_add_counter, EPSILON, EPSILON, *(return_state+returns), 0, 0);
+ j++;
+ *(handled_states2+(fsm2+i)->state_no) = 1;
+ if ((fsm2+i)->target != -1) {
+ add_fsm_arc(new_fsm, j, (fsm2+i)->state_no + state_add_counter, (fsm2+i)->in, (fsm2+i)->out , (fsm2+i)->target + state_add_counter, 0, 0);
+ j++;
+ }
+ } else {
+ add_fsm_arc(new_fsm, j, (fsm2+i)->state_no + state_add_counter, (fsm2+i)->in, (fsm2+i)->out, (fsm2+i)->target + state_add_counter, 0, 0);
+ j++;
+ }
+ }
+ state_add_counter = state_add_counter + states2;
+ }
+
+ add_fsm_arc(new_fsm, j, -1, -1, -1, -1, -1, -1);
+ xxfree(handled_states1);
+ xxfree(handled_states2);
+ xxfree(return_state);
+ xxfree(net1->states);
+ fsm_destroy(net2);
+ net1->states = new_fsm;
+ fsm_update_flags(net1, NO, NO, NO, NO, NO, NO);
+ fsm_count(net1);
+ return(net1);
+}
+
+/* Remove those symbols from sigma that have the same distribution as IDENTITY */
+
+void fsm_compact(struct fsm *net) {
+ struct checktable {
+ int state_no;
+ int target;
+ } *checktable;
+
+ struct fsm_state *fsm;
+ struct sigma *sig, *sigprev, *sign;
+ _Bool *potential;
+ int i, j, prevstate, numsymbols, in, out, state, target, removable;
+
+ fsm = net->states;
+ numsymbols = sigma_max(net->sigma);
+
+ potential = xxmalloc(sizeof(_Bool)*(numsymbols+1));
+ checktable = xxmalloc(sizeof(struct checktable)*(numsymbols+1));
+
+ for (i=0; i <= numsymbols; i++) {
+ *(potential+i) = 1;
+ (checktable+i)->state_no = -1;
+ (checktable+i)->target = -1;
+ }
+ /* For consistency reasons, can't remove symbols longer than 1 */
+ /* since @ and ? only match utf8 symbols of length 1 */
+
+ for (sig = net->sigma; sig != NULL && sig->number != -1; sig = sig->next) {
+ if (utf8strlen(sig->symbol) > 1) {
+ *(potential+sig->number) = 0;
+ }
+ }
+
+ prevstate = 0;
+
+ for (i=0; ; i++) {
+
+ if ((fsm+i)->state_no != prevstate) {
+ for (j=3; j<=numsymbols;j++) {
+ if ((checktable+j)->state_no != prevstate && (checktable+IDENTITY)->state_no != prevstate) {
+ continue;
+ }
+ if ((checktable+j)->target == (checktable+IDENTITY)->target && (checktable+j)->state_no == (checktable+IDENTITY)->state_no) {
+ continue;
+ }
+ *(potential+j) = 0;
+ }
+ }
+
+ if ((fsm+i)->state_no == -1)
+ break;
+
+ in = (fsm+i)->in;
+ out = (fsm+i)->out;
+ state = (fsm+i)->state_no;
+ target = (fsm+i)->target;
+
+ if (in != -1 && out != -1) {
+ if (((in == out && in > 2) || in == IDENTITY)) {
+ (checktable+in)->state_no = state;
+ (checktable+in)->target = target;
+ }
+ if (in != out && in > 2) {
+ *(potential+in) = 0;
+ }
+ if (in != out && out > 2) {
+ *(potential+out) = 0;
+ }
+ }
+ prevstate = state;
+ }
+ for (removable = 0, i=3; i <= numsymbols; i++) {
+ if (*(potential+i) == 1) {
+ removable = 1;
+ }
+
+ }
+ if (removable == 0) {
+ xxfree(potential);
+ xxfree(checktable);
+ return;
+ }
+ i = j = 0;
+ do {
+ in = (fsm+i)->in;
+
+ add_fsm_arc(fsm, j ,(fsm+i)->state_no,(fsm+i)->in,(fsm+i)->out,(fsm+i)->target,(fsm+i)->final_state,(fsm+i)->start_state);
+ if (in == -1) {
+ i++;
+ j++;
+ }
+ else if (*(potential+in) == 1 && in > 2) {
+ i++;
+ } else {
+ i++;
+ j++;
+ }
+ } while ((fsm+i)->state_no != -1);
+ add_fsm_arc(fsm, j ,(fsm+i)->state_no,(fsm+i)->in,(fsm+i)->out,(fsm+i)->target,(fsm+i)->final_state,(fsm+i)->start_state);
+
+ sigprev = NULL;
+ for (sig = net->sigma; sig != NULL && sig->number != -1; sig = sign) {
+
+ if ((sig->number > 2) && (*(potential+sig->number) == 1)) {
+ sigprev->next = sig->next;
+ sign = sig->next;
+ xxfree(sig->symbol);
+ xxfree(sig);
+ } else {
+ sigprev = sig;
+ sign = sig->next;
+ }
+ }
+ xxfree(potential);
+ xxfree(checktable);
+ sigma_cleanup(net,0);
+}
+
+int fsm_symbol_occurs(struct fsm *net, char *symbol, int side) {
+ struct fsm_state *fsm;
+ int i, sym;
+ sym = sigma_find(symbol, net->sigma);
+ if (sym == -1) {
+ return 0;
+ }
+ for (i=0, fsm = net->states; (fsm+i)->state_no != -1; i++) {
+ if (side == M_UPPER && (fsm+i)->in == sym)
+ return 1;
+ if (side == M_LOWER && (fsm+i)->out == sym)
+ return 1;
+ if (side == (M_UPPER + M_LOWER) && ( (fsm+i)->in == sym || (fsm+i)->out == sym))
+ return 1;
+ }
+ return 0;
+}
+
+struct fsm *fsm_equal_substrings(struct fsm *net, struct fsm *left, struct fsm *right) {
+
+ /* The algorithm extracts from the lower side all and only those strings where */
+ /* every X occurring in different substrings ... left X right ... is identical. */
+
+ /* Caveat: there is no reliable termination condition for the loop that extracts */
+ /* identities. This means that if run on languages where there are potentially */
+ /* infinite-length identical delimited substrings, it will not terminate. */
+
+ /* For example: _eq(l a* r l a* r, l , r) will not terminate. */
+
+ /* However, even if the languages occuring between left and right are infinite */
+ /* the algorithm terminates eventually if the the possible combinations of */
+ /* identical substrings is finite in length. */
+
+ /* For example _eq([l a* b r l a b* r]*, l, r) does terminate even though */
+ /* it contains a potentially infinite number of delimited substrings since */
+ /* the maximum length of the possible identical delimited substrings is finite. */
+
+ /* In this case the above example evaluates to the language [l a b r l a b r]* */
+
+ /* The algorithm: */
+ /* Input: L, left, right */
+ /* Output: the language that preserves only those strings in L */
+ /* where X is the same for all left X right sequences */
+
+ /* 1. mark all instances of left with ... LB and right with RB ... in L */
+
+ /* 2. split L into Leq and Lbypass */
+ /* where Lbypass are all those strings where LB RB sequences aren't */
+ /* properly nested (we must have ... LB ... RB ... LB ... RB ... etc.) */
+ /* Lbypass also includes all those with less than two bracketed */
+ /* instances, since we don't need to check equality if there's only */
+ /* one bracketed substring. */
+ /* We also remove the auxiliary LB and RB symbols from Lbypass */
+
+ /* 3. We extract all the possible symbols occurring between LB and RB */
+ /* from Leq */
+
+ /* 4. We create the transducer Move from all symbols in (3) */
+ /* Move = M(sym_1) | ... | M(sym_n) */
+ /* where M(a) is defined as [\LB* LB:a a:LB]* \LB* */
+ /* i.e. it rewrites bracketed strings such as "LB a b RB LB a b RB" */
+ /* to "a LB b RB a LB b RB" in effect moving brackets to the right */
+ /* one step for a symbol. */
+
+ /* 5. Leq = Cleanup(Leq) */
+ /* Cleanup removes LB RB sequences and, at the same time filters out */
+ /* any strings where we find both LB RB and LB X RB where X is not 0. */
+ /* since we know such sequences could not possibly be identical */
+ /* Cleanup is implemented by composing Leq with */
+ /* \LB* [LB:0 RB:0 \LB*]* | ~$[LB RB] */
+ /* - if the symbol LB does not occur on the lower side of Leq, goto(6) */
+ /* - else Leq = Move(Leq), goto(5) */
+
+ /* 6. Result = L .o. [Leq | Lbypass] */
+
+ int syms;
+ struct sigma *sig;
+ struct fsm *LB, *RB, *NOLB, *NORB, *InsertBrackets, *RemoveBrackets, *Lbracketed, *NOBR, *BracketFilter, *Lbypass, *Leq, *Labels, *Cleanup, *ThisMove, *ThisSymbol, *Move, *Result, *oldnet;
+
+ oldnet = fsm_copy(net);
+
+ /* LB = "@<eq<@" */
+ /* RB = "@>eq>@" */
+
+ LB = fsm_symbol("@<eq<@");
+ NOLB = fsm_minimize(fsm_term_negation(fsm_copy(LB)));
+ RB = fsm_symbol("@>eq>@");
+ NORB = fsm_minimize(fsm_term_negation(fsm_copy(RB)));
+ /* NOBR = ~$[LB|RB] */
+ NOBR = fsm_minimize(fsm_complement(fsm_contains(fsm_union(fsm_copy(LB),fsm_copy(RB)))));
+
+ sigma_add("@<eq<@", net->sigma);
+ sigma_add("@>eq>@", net->sigma);
+ sigma_sort(net);
+
+ /* Insert our aux markers into the language */
+
+ /* InsertBrackets = [~$[L|R] [L 0:LB|0:RB R]]* ~$[L|R]; */
+
+ InsertBrackets = fsm_minimize(fsm_concat(fsm_kleene_star(fsm_concat(fsm_complement(fsm_contains(fsm_union(fsm_copy(left),fsm_copy(right)))),fsm_union(fsm_concat(fsm_copy(left),fsm_cross_product(fsm_empty_string(),fsm_copy(LB))),fsm_concat(fsm_cross_product(fsm_empty_string(),fsm_copy(RB)),fsm_copy(right))))),fsm_complement(fsm_contains(fsm_union(fsm_copy(left),fsm_copy(right))))));
+
+
+ /* Lbracketed = L .o. InsertBrackets */
+
+ Lbracketed = fsm_compose(fsm_copy(net), InsertBrackets);
+
+ /* Filter out improper nestings, or languages with less than two marker pairs */
+
+ /* BracketFilter = NOBR LB NOBR RB NOBR [LB NOBR RB NOBR]+ */
+
+ BracketFilter = fsm_concat(fsm_copy(NOBR),fsm_concat(fsm_copy(LB),fsm_concat(fsm_copy(NOBR),fsm_concat(fsm_copy(RB),fsm_concat(fsm_copy(NOBR),fsm_kleene_plus(fsm_concat(fsm_copy(LB),fsm_concat(fsm_copy(NOBR),fsm_concat(fsm_copy(RB),fsm_copy(NOBR))))))))));
+
+ /* RemoveBrackets = [LB:0|RB:0|NOBR]* */
+ /* Lbypass = [Lbracketed .o. ~BracketFilter .o. LB|RB -> 0] */
+ /* Leq = [Lbracketed .o. BracketFilter] */
+
+ RemoveBrackets = fsm_kleene_star(fsm_union(fsm_cross_product(fsm_copy(LB),fsm_empty_string()),fsm_union(fsm_cross_product(fsm_copy(RB),fsm_empty_string()),fsm_copy(NOBR))));
+
+
+ Lbypass = fsm_lower(fsm_compose(fsm_copy(Lbracketed),fsm_compose(fsm_complement(fsm_copy(BracketFilter)),RemoveBrackets)));
+ Leq = fsm_compose(Lbracketed, BracketFilter);
+
+ /* Extract labels from lower side of L */
+ /* [Leq .o. [\LB:0* LB:0 \RB* RB:0]* \LB:0*].l */
+
+ Labels = fsm_sigma_pairs_net(fsm_lower(fsm_compose(fsm_copy(Leq),fsm_concat(fsm_kleene_star(fsm_concat(fsm_kleene_star(fsm_cross_product(fsm_copy(NOLB),fsm_empty_string())),fsm_concat(fsm_cross_product(fsm_copy(LB),fsm_empty_string()),fsm_concat(fsm_kleene_star(fsm_copy(NORB)),fsm_cross_product(fsm_copy(RB),fsm_empty_string()))))),fsm_kleene_star(fsm_cross_product(fsm_copy(NOLB),fsm_empty_string()))))));
+
+ /* Cleanup = \LB* [LB:0 RB:0 \LB*]* | ~$[LB RB] */
+
+ Cleanup = fsm_minimize(fsm_union(fsm_concat(fsm_kleene_star(fsm_copy(NOLB)),fsm_kleene_star(fsm_concat(fsm_cross_product(fsm_copy(LB),fsm_empty_string()),fsm_concat(fsm_cross_product(fsm_copy(RB),fsm_empty_string()),fsm_kleene_star(fsm_copy(NOLB)))))),fsm_complement(fsm_contains(fsm_concat(fsm_copy(LB),fsm_copy(RB))))));
+
+ /* Construct the move function */
+
+ Move = fsm_empty_string();
+
+ syms = 0;
+ for (sig = Labels->sigma; sig != NULL; sig = sig->next) {
+ /* Unclear which is faster: the first or the second version */
+ /* ThisMove = [\LB* LB:X X:LB]* \LB* */
+ /* ThisMove = [\LB* LB:0 X 0:LB]* \LB* */
+ if (sig->number >= 3) {
+ ThisSymbol = fsm_symbol(sig->symbol);
+ //ThisMove = fsm_concat(fsm_kleene_star(fsm_concat(fsm_kleene_star(fsm_copy(NOLB)),fsm_concat(fsm_cross_product(fsm_copy(LB),fsm_copy(ThisSymbol)),fsm_cross_product(fsm_copy(ThisSymbol),fsm_copy(LB))))), fsm_kleene_star(fsm_copy(NOLB)));
+ ThisMove = fsm_concat(fsm_kleene_star(fsm_concat(fsm_kleene_star(fsm_copy(NOLB)),fsm_concat(fsm_cross_product(fsm_copy(LB),fsm_empty_string()), fsm_concat(fsm_copy(ThisSymbol), fsm_cross_product(fsm_empty_string(),fsm_copy(LB)))))), fsm_kleene_star(fsm_copy(NOLB)));
+
+ Move = fsm_union(Move, ThisMove);
+ syms++;
+ }
+ }
+ Move = fsm_minimize(Move);
+ if (syms == 0) {
+ //printf("no syms");
+ fsm_destroy(net);
+ return(oldnet);
+ }
+
+ /* Move until no bracket symbols remain */
+ for (;;) {
+ //printf("Zapping\n");
+ Leq = fsm_compose(Leq, fsm_copy(Cleanup));
+ if (!fsm_symbol_occurs(Leq, "@<eq<@", M_LOWER))
+ break;
+ Leq = fsm_compose(Leq, fsm_copy(Move));
+ //Leq = fsm_minimize(fsm_compose(Leq, fsm_copy(Move)));
+ // printf("size: %i\n",Leq->statecount);
+ }
+
+ /* Result = L .o. [Leq | Lbypass] */
+ Result = fsm_minimize(fsm_compose(net, fsm_union(fsm_lower(Leq), Lbypass)));
+ sigma_remove("@<eq<@", Result->sigma);
+ sigma_remove("@>eq>@", Result->sigma);
+ fsm_compact(Result);
+ sigma_sort(Result);
+ fsm_destroy(oldnet);
+ return(Result);
+}
+
+struct fsm *fsm_invert(struct fsm *net) {
+ struct fsm_state *fsm;
+ int i, temp;
+
+ fsm = net->states;
+ for (i = 0; (fsm+i)->state_no != -1; i++) {
+ temp = (fsm+i)->in;
+ (fsm+i)->in = (fsm+i)->out;
+ (fsm+i)->out = temp;
+ }
+ i = net->arcs_sorted_in;
+ net->arcs_sorted_in = net->arcs_sorted_out;
+ net->arcs_sorted_out = i;
+ return (net);
+}
+
+struct fsm *fsm_sequentialize(struct fsm *net) {
+ printf("Implementation pending\n");
+ return(net);
+}
+
+
+struct fsm *fsm_bimachine(struct fsm *net) {
+ printf("implementation pending\n");
+ return(net);
+}
+
+/* _leftrewr(L, a:b) does a -> b || .#. L _ */
+/* _leftrewr(?* L, a:b) does a -> b || L _ */
+/* works only with single symbols, but is fast */
+
+struct fsm *fsm_left_rewr(struct fsm *net, struct fsm *rewr) {
+ struct fsm_construct_handle *outh;
+ struct fsm_read_handle *inh;
+ struct fsm *newnet;
+ int i, maxsigma, *sigmatable, currstate, sinkstate, seensource, innum, outnum, relabelin, relabelout, addedsink;
+
+ fsm_merge_sigma(net, rewr);
+ relabelin = rewr->states->in;
+ relabelout = rewr->states->out;
+
+ inh = fsm_read_init(net);
+ sinkstate = fsm_get_num_states(inh);
+ outh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(outh, net->sigma);
+ maxsigma = sigma_max(net->sigma);
+ maxsigma++;
+ sigmatable = xxmalloc(maxsigma * sizeof(int));
+ for (i = 0; i < maxsigma; i++) {
+ *(sigmatable+i) = -1;
+ }
+ addedsink = 0;
+ while ((currstate = fsm_get_next_state(inh)) != -1) {
+ seensource = 0;
+ fsm_construct_set_final(outh, currstate);
+
+ while (fsm_get_next_state_arc(inh)) {
+ innum = fsm_get_arc_num_in(inh);
+ outnum = fsm_get_arc_num_out(inh);
+ *(sigmatable+innum) = currstate;
+ if (innum == relabelin) {
+ seensource = 1;
+ if (fsm_read_is_final(inh, currstate)) {
+ outnum = relabelout;
+ }
+ }
+ fsm_construct_add_arc_nums(outh, fsm_get_arc_source(inh), fsm_get_arc_target(inh), innum, outnum);
+ }
+ for (i = 2; i < maxsigma; i++) {
+ if (*(sigmatable+i) != currstate && i != relabelin) {
+ fsm_construct_add_arc_nums(outh, currstate, sinkstate, i, i);
+ addedsink = 1;
+ }
+ }
+ if (seensource == 0) {
+ addedsink = 1;
+ if (fsm_read_is_final(inh, currstate)) {
+ fsm_construct_add_arc_nums(outh, currstate, sinkstate, relabelin, relabelout);
+ } else {
+ fsm_construct_add_arc_nums(outh, currstate, sinkstate, relabelin, relabelin);
+ }
+ }
+ }
+ if (addedsink) {
+ for (i = 2; i < maxsigma; i++) {
+ fsm_construct_add_arc_nums(outh, sinkstate, sinkstate, i, i);
+ }
+ fsm_construct_set_final(outh, sinkstate);
+ }
+ fsm_construct_set_initial(outh, 0);
+ fsm_read_done(inh);
+ newnet = fsm_construct_done(outh);
+ xxfree(sigmatable);
+ fsm_destroy(net);
+ fsm_destroy(rewr);
+ return(newnet);
+}
+
+struct fsm *fsm_add_sink(struct fsm *net, int final) {
+ struct fsm_construct_handle *outh;
+ struct fsm_read_handle *inh;
+ struct fsm *newnet;
+ int i, maxsigma, *sigmatable, currstate, sinkstate;
+
+ inh = fsm_read_init(net);
+ sinkstate = fsm_get_num_states(inh);
+ outh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(outh, net->sigma);
+ maxsigma = sigma_max(net->sigma);
+ maxsigma++;
+ sigmatable = xxmalloc(maxsigma * sizeof(int));
+ for (i = 0; i < maxsigma; i++) {
+ *(sigmatable+i) = -1;
+ }
+ while ((currstate = fsm_get_next_state(inh)) != -1) {
+ while (fsm_get_next_state_arc(inh)) {
+ fsm_construct_add_arc_nums(outh, fsm_get_arc_source(inh), fsm_get_arc_target(inh), fsm_get_arc_num_in(inh), fsm_get_arc_num_out(inh));
+ *(sigmatable+fsm_get_arc_num_in(inh)) = currstate;
+ }
+ for (i = 2; i < maxsigma; i++) {
+ if (*(sigmatable+i) != currstate) {
+ fsm_construct_add_arc_nums(outh, currstate, sinkstate, i, i);
+ }
+ }
+ }
+ for (i = 2; i < maxsigma; i++) {
+ fsm_construct_add_arc_nums(outh, sinkstate, sinkstate, i, i);
+ }
+
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(outh, i);
+ }
+ if (final == 1) {
+ fsm_construct_set_final(outh, sinkstate);
+ }
+ fsm_construct_set_initial(outh, 0);
+ fsm_read_done(inh);
+ newnet = fsm_construct_done(outh);
+ fsm_destroy(net);
+ return(newnet);
+}
+
+/* _addfinalloop(L, "#":0) adds "#":0 at all final states */
+/* _addnonfinalloop(L, "#":0) adds "#":0 at all nonfinal states */
+/* _addloop(L, "#":0) adds "#":0 at all states */
+
+/* Adds loops at finals = 0 nonfinals, finals = 1 finals, finals = 2, all */
+
+struct fsm *fsm_add_loop(struct fsm *net, struct fsm *marker, int finals) {
+ struct fsm *newnet;
+ struct fsm_construct_handle *outh;
+ struct fsm_read_handle *inh, *minh;
+ int i;
+
+ inh = fsm_read_init(net);
+ minh = fsm_read_init(marker);
+
+ outh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(outh, net->sigma);
+
+ while (fsm_get_next_arc(inh)) {
+ fsm_construct_add_arc_nums(outh, fsm_get_arc_source(inh), fsm_get_arc_target(inh), fsm_get_arc_num_in(inh), fsm_get_arc_num_out(inh));
+ }
+ /* Where to put the loops */
+ if (finals == 1) {
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(outh, i);
+ fsm_read_reset(minh);
+ while (fsm_get_next_arc(minh)) {
+ fsm_construct_add_arc(outh, i, i, fsm_get_arc_in(minh), fsm_get_arc_out(minh));
+ }
+ }
+ } else if (finals == 0 || finals == 2) {
+ for (i=0; i < net->statecount; i++) {
+ if (finals == 2 || !fsm_read_is_final(inh, i)) {
+ fsm_read_reset(minh);
+ while (fsm_get_next_arc(minh)) {
+ fsm_construct_add_arc(outh, i, i, fsm_get_arc_in(minh), fsm_get_arc_out(minh));
+ }
+ }
+ }
+ }
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(outh, i);
+ }
+ fsm_construct_set_initial(outh, 0);
+ fsm_read_done(inh);
+ fsm_read_done(minh);
+ newnet = fsm_construct_done(outh);
+ fsm_destroy(net);
+ return(newnet);
+}
+
+/* _marktail(?* L, 0:x) does ~$x .o. [..] -> x || L _ ; */
+/* _marktail(?* R.r, 0:x).r does ~$x .o. [..] -> x || _ R */
+
+struct fsm *fsm_mark_fsm_tail(struct fsm *net, struct fsm *marker) {
+ struct fsm *newnet;
+ struct fsm_construct_handle *outh;
+ struct fsm_read_handle *inh, *minh;
+ int i, *mappings, maxstate, target, newtarget;
+
+ inh = fsm_read_init(net);
+ minh = fsm_read_init(marker);
+
+ outh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(outh, net->sigma);
+
+ mappings = xxcalloc(net->statecount, sizeof(int));
+ maxstate = net->statecount;
+
+ while (fsm_get_next_arc(inh)) {
+ target = fsm_get_arc_target(inh);
+ if (fsm_read_is_final(inh, target)) {
+ if (!*(mappings+target)) {
+ newtarget = maxstate;
+ *(mappings+target) = newtarget;
+ fsm_read_reset(minh);
+ while (fsm_get_next_arc(minh)) {
+ fsm_construct_add_arc(outh, newtarget, target, fsm_get_arc_in(minh), fsm_get_arc_out(minh));
+ }
+ maxstate++;
+ } else {
+ newtarget = *(mappings+target);
+ }
+ fsm_construct_add_arc_nums(outh, fsm_get_arc_source(inh), newtarget, fsm_get_arc_num_in(inh), fsm_get_arc_num_out(inh));
+ } else {
+ fsm_construct_add_arc_nums(outh, fsm_get_arc_source(inh), target, fsm_get_arc_num_in(inh), fsm_get_arc_num_out(inh));
+ }
+ }
+ for (i=0; i < net->statecount; i++) {
+ fsm_construct_set_final(outh,i);
+ }
+
+ fsm_construct_set_initial(outh, 0);
+ fsm_read_done(inh);
+ fsm_read_done(minh);
+ newnet = fsm_construct_done(outh);
+ fsm_destroy(net);
+ xxfree(mappings);
+ return(newnet);
+}
+
+struct fsm *fsm_context_restrict(struct fsm *X, struct fsmcontexts *LR) {
+
+ struct fsm *Var, *Notvar, *UnionL, *UnionP, *Result, *Word;
+ struct fsmcontexts *pairs;
+
+ /* [.#. \.#.* .#.]-`[[ [\X* X C X \X*]&~[\X* [L1 X \X* X R1|...|Ln X \X* X Rn] \X*]],X,0] */
+ /* Where X = variable symbol */
+ /* The above only works if we do the subtraction iff the right hand side contains .#. in */
+ /* its alphabet */
+ /* A more generic formula is the following: */
+
+ /* `[[[(?) \.#.* (?)] - `[[[\X* X C X \X*] - [\X* [L1 X \X* X R1|...|Ln X \X* X Rn] \X*] ],X,0],.#.,0]; */
+ /* Here, the LHS is another way of saying ~[?+ .#. ?+] */
+
+ Var = fsm_symbol("@VARX@");
+ Notvar = fsm_minimize(fsm_kleene_star(fsm_term_negation(fsm_symbol("@VARX@"))));
+
+ /* We add the variable symbol to all alphabets to avoid ? mathing it */
+ /* which would cause extra nondeterminism */
+ sigma_add("@VARX@", X->sigma);
+ sigma_sort(X);
+
+ /* Also, if any L or R is undeclared we add 0 */
+ for (pairs = LR; pairs != NULL; pairs = pairs->next) {
+ if (pairs->left == NULL) {
+ pairs->left = fsm_empty_string();
+ } else {
+ sigma_add("@VARX@",pairs->left->sigma);
+ sigma_substitute(".#.", "@#@", pairs->left->sigma);
+ sigma_sort(pairs->left);
+ }
+ if (pairs->right == NULL) {
+ pairs->right = fsm_empty_string();
+ } else {
+ sigma_add("@VARX@",pairs->right->sigma);
+ sigma_substitute(".#.", "@#@", pairs->right->sigma);
+ sigma_sort(pairs->right);
+ }
+ }
+
+ UnionP = fsm_empty_set();
+
+ for (pairs = LR; pairs != NULL ; pairs = pairs->next) {
+ UnionP = fsm_minimize(fsm_union(fsm_minimize(fsm_concat(fsm_copy(pairs->left),fsm_concat(fsm_copy(Var),fsm_concat(fsm_copy(Notvar),fsm_concat(fsm_copy(Var),fsm_copy(pairs->right)))))), UnionP));
+ }
+
+ UnionL = fsm_minimize(fsm_concat(fsm_copy(Notvar),fsm_concat(fsm_copy(Var), fsm_concat(fsm_copy(X), fsm_concat(fsm_copy(Var),fsm_copy(Notvar))))));
+
+ Result = fsm_intersect(UnionL, fsm_complement(fsm_concat(fsm_copy(Notvar),fsm_minimize(fsm_concat(fsm_copy(UnionP),fsm_copy(Notvar))))));
+ if (sigma_find("@VARX@", Result->sigma) != -1) {
+ Result = fsm_complement(fsm_substitute_symbol(Result, "@VARX@","@_EPSILON_SYMBOL_@"));
+ } else {
+ Result = fsm_complement(Result);
+ }
+
+ if (sigma_find("@#@", Result->sigma) != -1) {
+ Word = fsm_minimize(fsm_concat(fsm_symbol("@#@"),fsm_concat(fsm_kleene_star(fsm_term_negation(fsm_symbol("@#@"))),fsm_symbol("@#@"))));
+ Result = fsm_intersect(Word, Result);
+ Result = fsm_substitute_symbol(Result, "@#@", "@_EPSILON_SYMBOL_@");
+ }
+ fsm_destroy(UnionP);
+ fsm_destroy(Var);
+ fsm_destroy(Notvar);
+ fsm_destroy(X);
+ fsm_clear_contexts(pairs);
+ return(Result);
+}
+
+struct fsm *fsm_flatten(struct fsm *net, struct fsm *epsilon) {
+ struct fsm *newnet;
+ struct fsm_construct_handle *outh;
+ struct fsm_read_handle *inh, *eps;
+ int i, maxstate, in, out, target;
+ char *epssym, *instring, *outstring;
+
+ net = fsm_minimize(net);
+
+ inh = fsm_read_init(net);
+ eps = fsm_read_init(epsilon);
+ if (fsm_get_next_arc(eps) == -1) {
+ fsm_destroy(net);
+ fsm_destroy(epsilon);
+ return NULL;
+ }
+ epssym = strdup(fsm_get_arc_in(eps));
+ fsm_read_done(eps);
+
+ outh = fsm_construct_init(net->name);
+ maxstate = net->statecount;
+
+ fsm_construct_copy_sigma(outh, net->sigma);
+
+ while (fsm_get_next_arc(inh)) {
+ target = fsm_get_arc_target(inh);
+ in = fsm_get_arc_num_in(inh);
+ out = fsm_get_arc_num_out(inh);
+ if (in == EPSILON || out == EPSILON) {
+ instring = fsm_get_arc_in(inh);
+ outstring = fsm_get_arc_out(inh);
+ if (in == EPSILON) { instring = epssym; }
+ if (out == EPSILON) { outstring = epssym; }
+
+ fsm_construct_add_arc(outh, fsm_get_arc_source(inh), maxstate, instring, instring);
+ fsm_construct_add_arc(outh, maxstate, target, outstring, outstring);
+ } else {
+ fsm_construct_add_arc_nums(outh, fsm_get_arc_source(inh), maxstate, in, in);
+ fsm_construct_add_arc_nums(outh, maxstate, target, out, out);
+ }
+ maxstate++;
+ }
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(outh, i);
+ }
+ while ((i = fsm_get_next_initial(inh)) != -1) {
+ fsm_construct_set_initial(outh, i);
+ }
+
+ fsm_read_done(inh);
+ newnet = fsm_construct_done(outh);
+ fsm_destroy(net);
+ fsm_destroy(epsilon);
+ xxfree(epssym);
+ return(newnet);
+}
+
+/* Removes IDENTITY and UNKNOWN transitions. If mode = 1, only removes UNKNOWNs */
+struct fsm *fsm_close_sigma(struct fsm *net, int mode) {
+ struct fsm *newnet;
+ struct fsm_construct_handle *newh;
+ struct fsm_read_handle *inh;
+ int i;
+
+ inh = fsm_read_init(net);
+ newh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(newh, net->sigma);
+
+ while (fsm_get_next_arc(inh)) {
+ if ((fsm_get_arc_num_in(inh) != UNKNOWN && fsm_get_arc_num_in(inh) != IDENTITY && fsm_get_arc_num_out(inh) != UNKNOWN && fsm_get_arc_num_out(inh) != IDENTITY) || (mode == 1 && fsm_get_arc_num_in(inh) != UNKNOWN && fsm_get_arc_num_out(inh) != UNKNOWN))
+ fsm_construct_add_arc_nums(newh, fsm_get_arc_source(inh), fsm_get_arc_target(inh), fsm_get_arc_num_in(inh), fsm_get_arc_num_out(inh));
+ }
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_set_final(newh, i);
+ }
+ while ((i = fsm_get_next_initial(inh)) != -1) {
+ fsm_construct_set_initial(newh, i);
+ }
+ fsm_read_done(inh);
+ newnet = fsm_construct_done(newh);
+ fsm_destroy(net);
+ return(fsm_minimize(newnet));
+}
diff --git a/contrib/foma2js.perl b/contrib/foma2js.perl
new file mode 100755
index 0000000..5e55281
--- /dev/null
+++ b/contrib/foma2js.perl
@@ -0,0 +1,173 @@
+#!/usr/bin/perl
+
+###################################################################
+# Converts foma file to js array for use with Javascript runtime #
+# Outputs a js array of all the transitions, indexed in the #
+# input direction. This array can be passed to the js function #
+# foma_apply_down() in foma_apply_down.js for stand-alone #
+# transducer application. #
+# #
+# Usage: foma2js [-n array variable name] [file] #
+# MH 20120127 #
+###################################################################
+
+use Switch;
+use utf8;
+use Compress::Zlib;
+
+my $buffer ; my $filein ; my $file; my $jsnetname = 'myNet';
+
+die "Usage: fomatojs [-n name] filename" if $#ARGV < 0;
+foreach (my $argnum = 0 ; $argnum <= $#ARGV; $argnum++) {
+ if ($ARGV[$argnum] =~ '^-h$') {
+ print "Usage: foma2js [-n array variable name] [file]\n";
+ exit;
+ }
+ if ($ARGV[$argnum] =~ '^-n$') {
+ if ($argnum+1 <= $#ARGV) {
+ $jsnetname = $ARGV[$argnum+1];
+ } else {
+ die;
+ }
+ } else {
+ $file = $ARGV[$argnum];
+ }
+}
+
+my $gz = gzopen($file, "rb")
+ or die "Cannot open $file: $gzerrno\n" ;
+while ($gz->gzread($buffer) > 0) {
+ $filein .= $buffer;
+}
+
+die "Error reading from $file: $gzerrno" . ($gzerrno+0) . "\n"
+ if $gzerrno != Z_STREAM_END ;
+$gz->gzclose();
+
+my @lines = split '\n', $filein;
+
+my $mode = "none";
+my $version; my $numnets = 0; my %pr;
+my $line = 0;
+my @sigma;
+my $longestsymbollength = 0;
+foreach (@lines) {
+ chomp;
+ if ($_ =~ /##foma-net ([0-9]+\.[0-9]+)##/) {
+ $version = $1;
+ $numnets++;
+ if ($numnets > 1) {
+ die "Only one network per file supported"
+ }
+ next;
+ }
+ if ($_ =~ /##props##/) {
+ $mode = "props";
+ next;
+ }
+ if ($_ =~ /##sigma##/) {
+ $mode = "sigma";
+ next;
+ }
+ if ($_ =~ /##states##/) {
+ $mode = "states";
+ next;
+ }
+ if ($_ =~ /##end##/) {
+ $mode = "none";
+ next;
+ }
+ switch($mode) {
+ case "props" {
+ ($pr{"arity"}, $pr{"arccount"},$pr{"statecount"},$pr{"linecount"}, $pr{"finalcount"},$pr{"pathcount"},$pr{"is_deterministic"},$pr{"is_pruned"},$pr{"is_minimized"},$pr{"is_epsilon_free"},$pr{"is_loop_free"},$pr{"extras"},$pr{"name"}) = split ' ';
+ }
+ case "states" {
+ #state in out target final
+ @transitions = split ' ';
+ if ($transitions[0] == -1) { next; }
+ if ($transitions[1] == -1 && $#transitions == 3) {
+ $arrstate = $transitions[0];
+ $arrfinal = $transitions[3];
+ if ($arrfinal == 1) {
+ $finals[$arrstate] = 1;
+ }
+ next;
+ }
+ if ($#transitions == 4) {
+ $arrstate = $transitions[0];
+ $arrin = $transitions[1];
+ $arrout = $transitions[2];
+ $arrtarget = $transitions[3];
+ $arrfinal = $transitions[4];
+ if ($arrfinal == 1) {
+ $finals[$arrstate] = 1;
+ }
+ }
+ elsif ($#transitions == 3) {
+ $arrstate = $transitions[0];
+ $arrin = $transitions[1];
+ $arrtarget = $transitions[2];
+ $arrfinal = $transitions[3];
+ $arrout = $arrin;
+ if ($arrfinal == 1) {
+ $finals[$arrstate] = 1;
+ }
+ }
+ elsif ($#transitions == 2) {
+ $arrin = $transitions[0];
+ $arrout = $transitions[1];
+ $arrtarget = $transitions[2];
+ }
+ elsif ($#transitions == 1) {
+ $arrin = $transitions[0];
+ $arrtarget = $transitions[1];
+ $arrout = $arrin;
+ }
+ push(@{$trans{$arrstate ."|" .$sigma[$arrin]}}, "\{$arrtarget:\'$sigma[$arrout]\'\}");
+ }
+ case "sigma" {
+ (my $number, my $symbol) = split ' ';
+ $symbol =~ s/^\@_EPSILON_SYMBOL_\@$//g;
+ $symbol =~ s/^\@_IDENTITY_SYMBOL_\@$/\@ID\@/g;
+ $symbol =~ s/^\@_UNKNOWN_SYMBOL_\@$/\@UN\@/g;
+ $symbol =~ s/'/\\'/g;
+ $sigma[$number] = $symbol;
+ if ($number > 2) {
+ utf8::decode($symbol);
+ if (length($symbol) > $longestsymbollength) {
+ $longestsymbollength = length($symbol);
+ }
+
+ }
+ }
+ case "none" {
+ die "Format error";
+ }
+ }
+}
+
+print "var $jsnetname = new Object;\n";
+print "$jsnetname.t = Array;\n";
+print "$jsnetname.f = Array;\n";
+print "$jsnetname.s = Array;\n\n";
+
+foreach $k (keys %trans) {
+ ($state, $in) = split /\|/, $k;
+ $in =~ s/^\@UN\@$/\@ID\@/;
+ print "$jsnetname.t\[$state + '|' + \'$in\'\] = \[";
+ print join (',', @{$trans{$k}}) ."\];\n";
+}
+
+for ($i = 0; $i <= $pr{'statecount'}; $i++) {
+ if (defined $finals[$i] and $finals[$i]) {
+ print "$jsnetname.f\[$i\] = 1;\n";
+ }
+}
+
+for ($i = 3 ; $i <= $#sigma; $i++) {
+ if (defined $sigma[$i]) {
+ print "$jsnetname.s\['$sigma[$i]'\] = $i;\n";
+ }
+}
+
+print "$jsnetname.maxlen = $longestsymbollength ;\n";
diff --git a/contrib/foma_apply_down.js b/contrib/foma_apply_down.js
new file mode 100644
index 0000000..86ff374
--- /dev/null
+++ b/contrib/foma_apply_down.js
@@ -0,0 +1,49 @@
+// Basic recursive apply down function for Javascript runtime.
+// Caveat: does not support flag diacritics and will recurse infinitely
+// on input-side epsilon-loops.
+// Use the foma2js.perl script to convert foma binaries to a Javascript array which
+// is needed as the first argument of foma_apply_down.
+
+function foma_apply_down(Net, inString) {
+ Rep = new Object;
+ Rep.answer = new Array;
+ Rep.results = 0;
+ foma_apply_dn(Net, inString, 0, 0, '', Rep);
+ return(Rep.answer);
+}
+
+function foma_apply_dn(Net, inString, Position, State, outString, Reply) {
+ if (Net.f[State] === 1 && Position === inString.length) {
+ Reply.answer.push(outString);
+ Reply.results++;
+ }
+ var match = 0;
+ for (var len = 0; len <= Net.maxlen && len <= inString.length - Position; len++) {
+ var key = State + '|' + inString.substr(Position,len);
+ for (var key2 in Net.t[key]) {
+ for (var targetState in Net.t[key][key2]) {
+ if (targetState == null) {
+ return;
+ }
+ var outputSymbol = Net.t[key][key2][targetState];
+ match = 1;
+ if (outputSymbol === '@UN@') { outputSymbol = '?'; }
+ foma_apply_dn(Net, inString, Position+len, targetState, outString + outputSymbol, Reply);
+ }
+ }
+ }
+ if (match === 0 && Net.s[inString.substr(Position,1)] == null && inString.length > Position) {
+ key = State + '|' + '@ID@';
+ for (key2 in Net.t[key]) {
+ for (targetState in Net.t[key][key2]) {
+ if (targetState == null) {
+ return;
+ }
+ outputSymbol = Net.t[key][key2][targetState];
+ if (outputSymbol === '@UN@') { outputSymbol = '?'; }
+ if (outputSymbol === '@ID@') { outputSymbol = inString.substr(Position,1); }
+ foma_apply_dn(Net, inString, Position+1, targetState, outString + outputSymbol, Reply);
+ }
+ }
+ }
+}
diff --git a/define.c b/define.c
new file mode 100644
index 0000000..a79e7f4
--- /dev/null
+++ b/define.c
@@ -0,0 +1,144 @@
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include "foma.h"
+
+/* Find a defined symbol from the symbol table */
+/* Return the corresponding FSM */
+struct fsm *find_defined(struct defined_networks *def, char *string) {
+ struct defined_networks *d;
+ for (d = def; d != NULL; d = d->next) {
+ if (d->name != NULL && strcmp(string, d->name) == 0) {
+ return(d->net);
+ }
+ }
+ return NULL;
+}
+
+struct defined_networks *defined_networks_init(void) {
+ struct defined_networks *def;
+ def = calloc(1, sizeof(struct defined_networks)); /* Dummy first entry, so we can maintain the ptr */
+ return def;
+}
+
+struct defined_functions *defined_functions_init(void) {
+ struct defined_functions *deff;
+ deff = calloc(1, sizeof(struct defined_functions)); /* Dummy first entry */
+ return deff;
+}
+
+/* Removes a defined network from the list */
+/* Returns 0 on success, 1 if the definition did not exist */
+/* Undefines all if NULL is passed as the string argument */
+
+int remove_defined(struct defined_networks *def, char *string) {
+ struct defined_networks *d, *d_prev, *d_next;
+ int exists = 0;
+ /* Undefine all */
+ if (string == NULL) {
+ for (d = def; d != NULL; d = d_next) {
+ d_next = d->next;
+ fsm_destroy(d->net);
+ xxfree(d->name);
+ }
+ return 0;
+ }
+ d_prev = NULL;
+ for (d = def; d != NULL; d_prev = d, d = d->next) {
+ if (d->name != NULL && strcmp(d->name, string) == 0) {
+ exists = 1;
+ break;
+ }
+ }
+ if (exists == 0) {
+ return 1;
+ }
+ if (d == def) {
+ if (d->next != NULL) {
+ fsm_destroy(d->net);
+ xxfree(d->name);
+ d->name = d->next->name;
+ d->net = d->next->net;
+ d_next = d->next->next;
+ xxfree(d->next);
+ d->next = d_next;
+ } else {
+ fsm_destroy(d->net);
+ xxfree(d->name);
+ d->next = NULL;
+ d->name = NULL;
+ }
+ } else {
+ fsm_destroy(d->net);
+ xxfree(d->name);
+ d_prev->next = d->next;
+ xxfree(d);
+ }
+ return 0;
+}
+
+/* Finds defined regex "function" based on name, numargs */
+/* Returns the corresponding regex */
+char *find_defined_function(struct defined_functions *deff, char *name, int numargs) {
+ struct defined_functions *d;
+ for (d = deff ; d != NULL; d = d->next) {
+ if (d->name != NULL && strcmp(d->name, name) == 0 && d->numargs == numargs) {
+ return(d->regex);
+ }
+ }
+ return NULL;
+}
+
+/* Add a function to list of defined functions */
+int add_defined_function(struct defined_functions *deff, char *name, char *regex, int numargs) {
+ struct defined_functions *d;
+ for (d = deff; d != NULL; d = d->next) {
+ if (d->name != NULL && strcmp(d->name, name) == 0 && d->numargs == numargs) {
+ xxfree(d->regex);
+ d->regex = xxstrdup(regex);
+ printf("redefined %s@%i)\n", name, numargs);
+ return 1;
+ }
+ }
+ if (deff->name == NULL) {
+ d = deff;
+ } else {
+ d = xxmalloc(sizeof(struct defined_functions));
+ d->next = deff->next;
+ deff->next = d;
+ }
+ d->name = xxstrdup(name);
+ d->regex = xxstrdup(regex);
+ d->numargs = numargs;
+ return 0;
+}
+
+/* Add a network to list of defined networks */
+/* Returns 0 on success or 1 on redefinition */
+/* Always maintain head of list at same ptr */
+
+int add_defined(struct defined_networks *def, struct fsm *net, char *string) {
+ struct defined_networks *d;
+ if (net == NULL)
+ return 0;
+ fsm_count(net);
+ for (d = def; d != NULL; d = d->next) {
+ if (d->name != NULL && strcmp(d->name, string) == 0) {
+ xxfree(d->net);
+ xxfree(d->name);
+ d->net = net;
+ d->name = xxstrdup(string);
+ return 1;
+ }
+ }
+ if (def->name == NULL) {
+ d = def;
+ } else {
+ d = xxmalloc(sizeof(struct defined_networks));
+ d->next = def->next;
+ def->next = d;
+ }
+ d->name = xxstrdup(string);
+ d->net = net;
+ return 0;
+}
diff --git a/determinize.c b/determinize.c
new file mode 100644
index 0000000..2498193
--- /dev/null
+++ b/determinize.c
@@ -0,0 +1,881 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+#include <limits.h>
+#include <stdint.h>
+#include <string.h>
+#include "foma.h"
+
+#define SUBSET_EPSILON_REMOVE 1
+#define SUBSET_DETERMINIZE 2
+#define SUBSET_TEST_STAR_FREE 3
+
+#define NHASH_LOAD_LIMIT 2 /* load limit for nhash table size */
+
+static int fsm_linecount, num_states, num_symbols, epsilon_symbol, *single_sigma_array, *double_sigma_array, limit, num_start_states, op;
+
+static _Bool *finals, deterministic, numss;
+
+struct e_closure_memo {
+ int state;
+ int mark;
+ struct e_closure_memo *target;
+ struct e_closure_memo *next;
+};
+
+static unsigned int primes[26] = {61,127,251,509,1021,2039,4093,8191,16381,32749,65521,131071,262139,524287,1048573,2097143,4194301,8388593,16777213,33554393,67108859,134217689,268435399,536870909,1073741789,2147483647};
+
+static struct e_closure_memo *e_closure_memo;
+
+int T_last_unmarked, T_limit;
+
+struct nhash_list {
+ int setnum;
+ unsigned int size;
+ unsigned int set_offset;
+ struct nhash_list *next;
+};
+
+struct T_memo {
+ unsigned char finalstart;
+ unsigned int size;
+ unsigned int set_offset;
+};
+
+struct trans_list {
+ int inout;
+ int target;
+} *trans_list;
+
+struct trans_array {
+ struct trans_list *transitions;
+ unsigned int size;
+ unsigned int tail;
+} *trans_array;
+
+static struct T_memo *T_ptr;
+
+static int nhash_tablesize, nhash_load, current_setnum, *e_table, *marktable, *temp_move, mainloop, maxsigma, *set_table, set_table_size, star_free_mark;
+static unsigned int set_table_offset;
+static struct nhash_list *table;
+
+extern int add_fsm_arc(struct fsm_state *fsm, int offset, int state_no, int in, int out, int target, int final_state, int start_state);
+
+static void init(struct fsm *net);
+INLINE static int e_closure(int states);
+INLINE static int set_lookup(int *lookup_table, int size);
+static int initial_e_closure(struct fsm *network);
+static void memoize_e_closure(struct fsm_state *fsm);
+static int next_unmarked(void);
+static void single_symbol_to_symbol_pair(int symbol, int *symbol_in, int *symbol_out);
+static int symbol_pair_to_single_symbol(int in, int out);
+static void sigma_to_pairs(struct fsm *net);
+static int nhash_find_insert(int *set, int setsize);
+INLINE static int hashf(int *set, int setsize);
+static int nhash_insert(int hashval, int *set, int setsize);
+static void nhash_rebuild_table ();
+static void nhash_init (int initial_size);
+static void nhash_free(struct nhash_list *nptr, int size);
+static void e_closure_free();
+static void init_trans_array(struct fsm *net);
+static struct fsm *fsm_subset(struct fsm *net, int operation);
+
+struct fsm *fsm_epsilon_remove(struct fsm *net) {
+ return(fsm_subset(net, SUBSET_EPSILON_REMOVE));
+}
+
+struct fsm *fsm_determinize(struct fsm *net) {
+ return(fsm_subset(net, SUBSET_DETERMINIZE));
+}
+
+int fsm_isstarfree(struct fsm *net) {
+ #define DFS_WHITE 0
+ #define DFS_GRAY 1
+ #define DFS_BLACK 2
+
+ struct fsm *sfnet;
+ struct state_array *state_array;
+ struct fsm_state *curr_ptr;
+ int v, vp, is_star_free;
+ short int in;
+ char *dfs_map;
+
+ sfnet = fsm_subset(net, SUBSET_TEST_STAR_FREE);
+ is_star_free = 1;
+
+ state_array = map_firstlines(net);
+ ptr_stack_clear();
+ ptr_stack_push(state_array->transitions);
+
+ dfs_map = xxcalloc(sfnet->statecount, sizeof(char));
+ while(!ptr_stack_isempty()) {
+
+ curr_ptr = ptr_stack_pop();
+ nopop:
+ v = curr_ptr->state_no; /* source state number */
+ vp = curr_ptr->target; /* target state number */
+
+ if (v == -1 || vp == -1) {
+ *(dfs_map+v) = DFS_BLACK;
+ continue;
+ }
+ *(dfs_map+v) = DFS_GRAY;
+
+ in = curr_ptr->in;
+ if (*(dfs_map+vp) == DFS_GRAY && in == maxsigma) {
+ /* Not star-free */
+ is_star_free = 0;
+ break;
+ }
+ if (v == (curr_ptr+1)->state_no) {
+ ptr_stack_push(curr_ptr+1);
+ }
+ if (*(dfs_map+vp) == DFS_WHITE) {
+ curr_ptr = (state_array+vp)->transitions;
+ goto nopop;
+ }
+ }
+ ptr_stack_clear();
+ xxfree(dfs_map);
+ xxfree(state_array);
+ //stack_add(sfnet);
+ return(is_star_free);
+}
+
+static struct fsm *fsm_subset(struct fsm *net, int operation) {
+
+ int T, U;
+
+ if (net->is_deterministic == YES && operation != SUBSET_TEST_STAR_FREE) {
+ return(net);
+ }
+ /* Export this var */
+ op = operation;
+ fsm_count(net);
+ num_states = net->statecount;
+ deterministic = 1;
+ init(net);
+ nhash_init((num_states < 12) ? 6 : num_states/2);
+
+ T = initial_e_closure(net);
+
+ int_stack_clear();
+
+ if (deterministic == 1 && epsilon_symbol == -1 && num_start_states == 1 && numss == 0) {
+ net->is_deterministic = YES;
+ net->is_epsilon_free = YES;
+ nhash_free(table, nhash_tablesize);
+ xxfree(T_ptr);
+ xxfree(e_table);
+ xxfree(trans_list);
+ xxfree(trans_array);
+ xxfree(double_sigma_array);
+ xxfree(single_sigma_array);
+ xxfree(finals);
+ xxfree(temp_move);
+ xxfree(set_table);
+ return(net);
+ }
+
+ if (operation == SUBSET_EPSILON_REMOVE && epsilon_symbol == -1) {
+ net->is_epsilon_free = YES;
+ nhash_free(table, nhash_tablesize);
+ xxfree(T_ptr);
+ xxfree(e_table);
+ xxfree(trans_list);
+ xxfree(trans_array);
+ xxfree(double_sigma_array);
+ xxfree(single_sigma_array);
+ xxfree(finals);
+ xxfree(temp_move);
+ xxfree(set_table);
+ return(net);
+ }
+
+ if (operation == SUBSET_TEST_STAR_FREE) {
+ fsm_state_init(sigma_max(net->sigma)+1);
+ star_free_mark = 0;
+ } else {
+ fsm_state_init(sigma_max(net->sigma));
+ xxfree(net->states);
+ }
+
+ /* init */
+
+ do {
+ int i, j, tail, setsize, *theset, stateno, has_trans, minsym, next_minsym, trgt, symbol_in, symbol_out;
+ struct trans_list *transitions;
+ struct trans_array *tptr;
+
+ fsm_state_set_current_state(T, (T_ptr+T)->finalstart, T == 0 ? 1 : 0);
+
+ /* Prepare set */
+ setsize = (T_ptr+T)->size;
+ theset = set_table+(T_ptr+T)->set_offset;
+ minsym = INT_MAX;
+ has_trans = 0;
+ for (i = 0; i < setsize; i++) {
+ stateno = *(theset+i);
+ tptr = trans_array+stateno;
+ tptr->tail = 0;
+ if (tptr->size == 0)
+ continue;
+ if ((tptr->transitions)->inout < minsym) {
+ minsym = (tptr->transitions)->inout;
+ has_trans = 1;
+ }
+ }
+ if (!has_trans) {
+ /* close state */
+ fsm_state_end_state();
+ continue;
+ }
+
+ /* While set not empty */
+
+ for (next_minsym = INT_MAX; minsym != INT_MAX ; minsym = next_minsym, next_minsym = INT_MAX) {
+ theset = set_table+(T_ptr+T)->set_offset;
+
+ for (i = 0, j = 0 ; i < setsize; i++) {
+
+ stateno = *(theset+i);
+ tptr = trans_array+stateno;
+ tail = tptr->tail;
+ transitions = (tptr->transitions)+tail;
+
+ while (tail < tptr->size && transitions->inout == minsym) {
+ trgt = transitions->target;
+ if (*(e_table+(trgt)) != mainloop) {
+ *(e_table+(trgt)) = mainloop;
+ *(temp_move+j) = trgt;
+ j++;
+
+ if (operation == SUBSET_EPSILON_REMOVE) {
+ mainloop++;
+ if ((U = e_closure(j)) != -1) {
+ single_symbol_to_symbol_pair(minsym, &symbol_in, &symbol_out);
+ fsm_state_add_arc(T, symbol_in, symbol_out, U, (T_ptr+T)->finalstart, T == 0 ? 1 : 0);
+ j = 0;
+ }
+ }
+ }
+ tail++;
+ transitions++;
+ }
+
+ tptr->tail = tail;
+
+ if (tail == tptr->size)
+ continue;
+ /* Check next minsym */
+ if (transitions->inout < next_minsym) {
+ next_minsym = transitions->inout;
+ }
+ }
+ if (operation == SUBSET_DETERMINIZE) {
+ mainloop++;
+ if ((U = e_closure(j)) != -1) {
+ single_symbol_to_symbol_pair(minsym, &symbol_in, &symbol_out);
+ fsm_state_add_arc(T, symbol_in, symbol_out, U, (T_ptr+T)->finalstart, T == 0 ? 1 : 0);
+ }
+ }
+ if (operation == SUBSET_TEST_STAR_FREE) {
+ mainloop++;
+ if ((U = e_closure(j)) != -1) {
+ single_symbol_to_symbol_pair(minsym, &symbol_in, &symbol_out);
+ fsm_state_add_arc(T, symbol_in, symbol_out, U, (T_ptr+T)->finalstart, T == 0 ? 1 : 0);
+ if (star_free_mark == 1) {
+ //fsm_state_add_arc(T, maxsigma, maxsigma, U, (T_ptr+T)->finalstart, T == 0 ? 1 : 0);
+ star_free_mark = 0;
+ }
+ }
+ }
+ }
+ /* end state */
+ fsm_state_end_state();
+ } while ((T = next_unmarked()) != -1);
+
+ /* wrapup() */
+ nhash_free(table, nhash_tablesize);
+ xxfree(set_table);
+ xxfree(T_ptr);
+ xxfree(temp_move);
+ xxfree(e_table);
+ xxfree(trans_list);
+ xxfree(trans_array);
+
+ if (epsilon_symbol != -1)
+ e_closure_free();
+ xxfree(double_sigma_array);
+ xxfree(single_sigma_array);
+ xxfree(finals);
+ fsm_state_close(net);
+ return(net);
+}
+
+static void init(struct fsm *net) {
+ /* A temporary table for handling epsilon closure */
+ /* to avoid doubles */
+
+ e_table = xxcalloc(net->statecount,sizeof(int));
+
+ /* Counter for our access tables */
+
+ mainloop = 1;
+
+ /* Temporary table for storing sets and */
+ /* passing to hash function */
+
+ /* Table for listing current results of move & e-closure */
+ temp_move = xxmalloc((net->statecount + 1) *sizeof(int));
+
+ /* We malloc this much memory to begin with for the new fsm */
+ /* Then grow it by the double as needed */
+
+ limit = next_power_of_two(net->linecount);
+ fsm_linecount = 0;
+ sigma_to_pairs(net);
+
+ /* Optimistically malloc T_ptr array */
+ /* We allocate memory for a number of pointers to a set of states */
+ /* To handle fast lookup in array */
+ /* Optimistically, we choose the initial size to be the number of */
+ /* states in the non-deterministic fsm */
+
+ T_last_unmarked = 0;
+ T_limit = next_power_of_two(num_states);
+
+ T_ptr = xxcalloc(T_limit,sizeof(struct T_memo));
+
+ /* Stores all sets consecutively in one table */
+ /* T_ptr->set_offset and size */
+ /* are used to retrieve the set */
+
+ set_table_size = next_power_of_two(num_states);
+ set_table = xxmalloc(set_table_size*sizeof(int));
+ set_table_offset = 0;
+
+ init_trans_array(net);
+}
+
+static int trans_sort_cmp(const void *a, const void *b) {
+ return (((const struct trans_list *)a)->inout - ((const struct trans_list *)b)->inout);
+}
+
+static void init_trans_array(struct fsm *net) {
+ struct trans_list *arrptr;
+ struct fsm_state *fsm;
+ int i, j, laststate, lastsym, inout, size, state;
+
+ arrptr = trans_list = xxmalloc(net->linecount * sizeof(struct trans_list));
+ trans_array = xxcalloc(net->statecount, sizeof(struct trans_array));
+
+ laststate = -1;
+ fsm = net->states;
+
+ for (i=0, size = 0; (fsm+i)->state_no != -1; i++) {
+ state = (fsm+i)->state_no;
+ if (state != laststate) {
+ if (laststate != -1) {
+ (trans_array+laststate)->size = size;
+ }
+ (trans_array+state)->transitions = arrptr;
+ size = 0;
+ }
+ laststate = state;
+
+ if ((fsm+i)->target == -1)
+ continue;
+ inout = symbol_pair_to_single_symbol((fsm+i)->in, (fsm+i)->out);
+ if (inout == epsilon_symbol)
+ continue;
+
+ arrptr->inout = inout;
+ arrptr->target = (fsm+i)->target;
+ arrptr++;
+ size++;
+ }
+
+ if (laststate != -1) {
+ (trans_array+laststate)->size = size;
+ }
+
+ for (i=0; i < net->statecount; i++) {
+ arrptr = (trans_array+i)->transitions;
+ size = (trans_array+i)->size;
+ if (size > 1) {
+ qsort(arrptr, size, sizeof(struct trans_list), trans_sort_cmp);
+ lastsym = -1;
+ /* Figure out if we're already deterministic */
+ for (j=0; j < size; j++) {
+ if ((arrptr+j)->inout == lastsym)
+ deterministic = 0;
+ lastsym = (arrptr+j)->inout;
+ }
+ }
+ }
+}
+
+INLINE static int e_closure(int states) {
+
+ int i, set_size;
+ struct e_closure_memo *ptr;
+
+ /* e_closure extends the list of states which are reachable */
+ /* and appends these to e_table */
+
+ if (epsilon_symbol == -1)
+ return(set_lookup(temp_move, states));
+
+ if (states == 0)
+ return -1;
+
+ mainloop--;
+
+ set_size = states;
+
+ for (i = 0; i < states; i++) {
+
+ /* State number we want to do e-closure on */
+ ptr = e_closure_memo + *(temp_move+i);
+ if (ptr->target == NULL)
+ continue;
+ ptr_stack_push(ptr);
+
+ while (!(ptr_stack_isempty())) {
+ ptr = ptr_stack_pop();
+ /* Don't follow if already seen */
+ if (*(marktable+ptr->state) == mainloop)
+ continue;
+
+ ptr->mark = mainloop;
+ *(marktable+ptr->state) = mainloop;
+ /* Add to tail of list */
+ if (*(e_table+(ptr->state)) != mainloop) {
+ *(temp_move+set_size) = ptr->state;
+ *(e_table+(ptr->state)) = mainloop;
+ set_size++;
+ }
+
+ if (ptr->target == NULL)
+ continue;
+ /* Traverse chain */
+
+ for (; ptr != NULL ; ptr = ptr->next) {
+ if (ptr->target->mark != mainloop) {
+ /* Push */
+ ptr->target->mark = mainloop;
+ ptr_stack_push(ptr->target);
+ }
+ }
+ }
+ }
+
+ mainloop++;
+ return(set_lookup(temp_move, set_size));
+}
+
+INLINE static int set_lookup (int *lookup_table, int size) {
+
+ /* Look up a set and its corresponding state number */
+ /* if it doesn't exist from before, assign a state number */
+
+ return(nhash_find_insert(lookup_table, size));
+
+}
+
+void add_T_ptr(int setnum, int setsize, unsigned int theset, int fs) {
+
+ int i;
+ if (setnum >= T_limit) {
+ T_limit *= 2;
+ T_ptr = xxrealloc(T_ptr, sizeof(struct T_memo)*T_limit);
+ for (i=setnum; i < T_limit; i++) {
+ (T_ptr+i)->size = 0;
+ }
+ }
+
+ (T_ptr + setnum)->size = setsize;
+ (T_ptr + setnum)->set_offset = theset;
+ (T_ptr + setnum)->finalstart = fs;
+ int_stack_push(setnum);
+
+}
+
+static int initial_e_closure(struct fsm *net) {
+
+ struct fsm_state *fsm;
+ int i,j;
+
+ finals = xxcalloc(num_states, sizeof(_Bool));
+
+ num_start_states = 0;
+ fsm = net->states;
+
+ /* Create lookups for each state */
+ for (i=0,j=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->final_state) {
+ finals[(fsm+i)->state_no] = 1;
+ }
+ /* Add the start states as the initial set */
+ if ((op == SUBSET_TEST_STAR_FREE) || ((fsm+i)->start_state)) {
+ if (*(e_table+((fsm+i)->state_no)) != mainloop) {
+ num_start_states++;
+ numss = (fsm+i)->state_no;
+ *(e_table+((fsm+i)->state_no)) = mainloop;
+ *(temp_move+j) = (fsm+i)->state_no;
+ j++;
+ }
+ }
+ }
+ mainloop++;
+ /* Memoize e-closure(u) */
+ if (epsilon_symbol != -1) {
+ memoize_e_closure(fsm);
+ }
+ return(e_closure(j));
+}
+
+static void memoize_e_closure(struct fsm_state *fsm) {
+
+ int i, state, laststate, *redcheck;
+ struct e_closure_memo *ptr;
+
+ e_closure_memo = xxcalloc(num_states,sizeof(struct e_closure_memo));
+ marktable = xxcalloc(num_states,sizeof(int));
+ /* Table for avoiding redundant epsilon arcs in closure */
+ redcheck = xxmalloc(num_states*sizeof(int));
+
+ for (i=0; i < num_states; i++) {
+ ptr = e_closure_memo+i;
+ ptr->state = i;
+ ptr->target = NULL;
+ *(redcheck+i) = -1;
+ }
+
+ laststate = -1;
+
+ for (i=0; ;i++) {
+
+ state = (fsm+i)->state_no;
+
+ if (state != laststate) {
+ if (!int_stack_isempty()) {
+ deterministic = 0;
+ ptr = e_closure_memo+laststate;
+ ptr->target = e_closure_memo+int_stack_pop();
+ while (!int_stack_isempty()) {
+ ptr->next = xxmalloc(sizeof(struct e_closure_memo));
+ ptr->next->state = laststate;
+ ptr->next->target = e_closure_memo+int_stack_pop();
+ ptr->next->next = NULL;
+ ptr = ptr->next;
+ }
+ }
+ }
+ if (state == -1) {
+ break;
+ }
+ if ((fsm+i)->target == -1) {
+ continue;
+ }
+ /* Check if we have a redundant epsilon arc */
+ if ((fsm+i)->in == EPSILON && (fsm+i)->out == EPSILON) {
+ if (*(redcheck+((fsm+i)->target)) != (fsm+i)->state_no) {
+ if ((fsm+i)->target != (fsm+i)->state_no) {
+ int_stack_push((fsm+i)->target);
+ *(redcheck+((fsm+i)->target)) = (fsm+i)->state_no;
+ }
+ }
+ laststate = state;
+ }
+ }
+ xxfree(redcheck);
+}
+
+static int next_unmarked(void) {
+ if ((int_stack_isempty()))
+ return -1;
+ return(int_stack_pop());
+
+ if ((T_limit <= T_last_unmarked + 1) || (T_ptr+T_last_unmarked+1)->size == 0) {
+ return -1;
+ } else {
+ T_last_unmarked++;
+ return(T_last_unmarked);
+ }
+}
+
+static void single_symbol_to_symbol_pair(int symbol, int *symbol_in, int *symbol_out) {
+
+ *symbol_in = *(single_sigma_array+(symbol*2));
+ *symbol_out = *(single_sigma_array+(symbol*2+1));
+
+}
+
+static int symbol_pair_to_single_symbol(int in, int out) {
+ return(*(double_sigma_array+maxsigma*in+out));
+}
+
+static void sigma_to_pairs(struct fsm *net) {
+
+ int i, j, x, y, z, next_x = 0;
+ struct fsm_state *fsm;
+
+ fsm = net->states;
+
+ epsilon_symbol = -1;
+ maxsigma = sigma_max(net->sigma);
+ maxsigma++;
+
+ single_sigma_array = xxmalloc(2*maxsigma*maxsigma*sizeof(int));
+ double_sigma_array = xxmalloc(maxsigma*maxsigma*sizeof(int));
+
+ for (i=0; i < maxsigma; i++) {
+ for (j=0; j< maxsigma; j++) {
+ *(double_sigma_array+maxsigma*i+j) = -1;
+ }
+ }
+
+ /* f(x) -> y,z sigma pair */
+ /* f(y,z) -> x simple entry */
+ /* if exists f(n) <-> EPSILON, EPSILON, save n */
+ /* symbol(x) x>=1 */
+
+ /* Forward mapping: */
+ /* *(double_sigma_array+maxsigma*in+out) */
+
+ /* Backmapping: */
+ /* *(single_sigma_array+(symbol*2) = in(symbol) */
+ /* *(single_sigma_array+(symbol*2+1) = out(symbol) */
+
+ /* Table for checking whether a state is final */
+
+ x = 0;
+ net->arity = 1;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ y = (fsm+i)->in;
+ z = (fsm+i)->out;
+ if ((y == -1) || (z == -1))
+ continue;
+ if (y != z || y == UNKNOWN || z == UNKNOWN)
+ net->arity = 2;
+ if (*(double_sigma_array+maxsigma*y+z) == -1) {
+ *(double_sigma_array+maxsigma*y+z) = x;
+ *(single_sigma_array+next_x) = y;
+ next_x++;
+ *(single_sigma_array+next_x) = z;
+ next_x++;
+ if (y == EPSILON && z == EPSILON) {
+ epsilon_symbol = x;
+ }
+ x++;
+ }
+ }
+ num_symbols = x;
+}
+
+
+/* Functions for hashing n integers */
+/* with permutations hashing to the same value */
+/* necessary for subset construction */
+
+static int nhash_find_insert(int *set, int setsize) {
+ int j, found, *currlist;
+ struct nhash_list *tableptr;
+ unsigned int hashval;
+
+ hashval = hashf(set, setsize);
+ if ((table+hashval)->size == 0) {
+ return(nhash_insert(hashval, set, setsize));
+ } else {
+ for (tableptr=(table+hashval); tableptr != NULL; tableptr = tableptr->next) {
+ if ((tableptr)->size != setsize) {
+ continue;
+ } else {
+ /* Compare the list at hashval position */
+ /* to the current set by looking at etable */
+ /* entries */
+ for (j=0, found = 1, currlist= set_table+tableptr->set_offset; j < setsize; j++) {
+ if (*(e_table+(*(currlist+j))) != (mainloop-1)) {
+ found = 0;
+ break;
+ }
+ }
+ if (op == SUBSET_TEST_STAR_FREE && found == 1) {
+ for (j=0, currlist= set_table+tableptr->set_offset; j < setsize; j++) {
+ if (*(set+j) != *(currlist+j)) {
+ /* Set mark */
+ star_free_mark = 1;
+ }
+ }
+ }
+ if (found == 1) {
+ return(tableptr->setnum);
+ }
+ }
+ }
+
+ if (nhash_load / NHASH_LOAD_LIMIT > nhash_tablesize) {
+ nhash_rebuild_table();
+ hashval = hashf(set, setsize);
+ }
+ return(nhash_insert(hashval, set, setsize));
+ }
+}
+
+INLINE static int hashf(int *set, int setsize) {
+ int i;
+ unsigned int hashval, sum = 0;
+ hashval = 6703271;
+ for (i = 0; i < setsize; i++) {
+ hashval = (unsigned int) (*(set+i) + 1103 * setsize) * hashval;
+ sum += *(set+i) + i;
+ }
+ hashval = hashval + sum * 31;
+ hashval = (hashval % nhash_tablesize);
+ return hashval;
+}
+
+static unsigned int move_set(int *set, int setsize) {
+ unsigned int old_offset;
+ if (set_table_offset + setsize >= set_table_size) {
+ while (set_table_offset + setsize >= set_table_size) {
+ set_table_size *= 2;
+ }
+ set_table = xxrealloc(set_table, set_table_size * sizeof(int));
+ }
+ memcpy(set_table+set_table_offset, set, setsize * sizeof(int));
+ old_offset = set_table_offset;
+ set_table_offset += setsize;
+ return(old_offset);
+}
+
+static int nhash_insert(int hashval, int *set, int setsize) {
+ struct nhash_list *tableptr;
+ int i, fs = 0;
+
+ current_setnum++;
+ tableptr = table+hashval;
+
+ nhash_load++;
+ for (i = 0; i < setsize; i++) {
+ if (finals[*(set+i)])
+ fs = 1;
+ }
+ if (tableptr->size == 0) {
+
+ tableptr->set_offset = move_set(set, setsize);
+ tableptr->size = setsize;
+ tableptr->setnum = current_setnum;
+
+ add_T_ptr(current_setnum, setsize, tableptr->set_offset, fs);
+ return(current_setnum);
+ }
+
+ tableptr = xxmalloc(sizeof(struct nhash_list));
+ tableptr->next = (table+hashval)->next;
+ (table+hashval)->next = tableptr;
+ tableptr->setnum = current_setnum;
+ tableptr->size = setsize;
+ tableptr->set_offset = move_set(set, setsize);
+
+ add_T_ptr(current_setnum, setsize, tableptr->set_offset, fs);
+ return(current_setnum);
+}
+
+static void nhash_rebuild_table () {
+ int i, oldsize;
+ struct nhash_list *oldtable, *tableptr, *ntableptr, *newptr;
+ unsigned int hashval;
+
+ oldtable = table;
+ oldsize = nhash_tablesize;
+
+ nhash_load = 0;
+ for (i=0; primes[i] < nhash_tablesize; i++) { }
+ nhash_tablesize = primes[(i+1)];
+
+ table = xxcalloc(nhash_tablesize,sizeof(struct nhash_list));
+ for (i=0; i < oldsize;i++) {
+ if ((oldtable+i)->size == 0) {
+ continue;
+ }
+ tableptr = oldtable+i;
+ for ( ; tableptr != NULL; (tableptr = tableptr->next)) {
+ /* rehash */
+ hashval = hashf(set_table+tableptr->set_offset,tableptr->size);
+ ntableptr = table+hashval;
+ if (ntableptr->size == 0) {
+ nhash_load++;
+ ntableptr->size = tableptr->size;
+ ntableptr->set_offset = tableptr->set_offset;
+ ntableptr->setnum = tableptr->setnum;
+ ntableptr->next = NULL;
+ } else {
+ newptr = xxmalloc(sizeof(struct nhash_list));
+ newptr->next = ntableptr->next;
+ ntableptr->next = newptr;
+ newptr->setnum = tableptr->setnum;
+ newptr->size = tableptr->size;
+ newptr->set_offset = tableptr->set_offset;
+ }
+ }
+ }
+ nhash_free(oldtable, oldsize);
+}
+
+static void nhash_init (int initial_size) {
+
+ int i;
+
+ for (i=0; primes[i] < initial_size; i++) { }
+ nhash_load = 0;
+ nhash_tablesize = primes[i];
+ table = xxcalloc(nhash_tablesize , sizeof(struct nhash_list));
+ current_setnum = -1;
+}
+static void e_closure_free() {
+ int i;
+ struct e_closure_memo *eptr, *eprev;
+ xxfree(marktable);
+ for (i=0;i < num_states; i++) {
+ eptr = (e_closure_memo+i)->next;
+ for (eprev = NULL; eptr != NULL; ) {
+ eprev = eptr;
+ eptr = eptr->next;
+ xxfree(eprev);
+ }
+
+ }
+ xxfree(e_closure_memo);
+}
+
+static void nhash_free(struct nhash_list *nptr, int size) {
+ struct nhash_list *nptr2, *nnext;
+ int i;
+ for (i=0; i < size; i++) {
+ for (nptr2 = (nptr+i)->next; nptr2 != NULL; nptr2 = nnext) {
+ nnext = nptr2->next;
+ xxfree(nptr2);
+ }
+ }
+ xxfree(nptr);
+}
diff --git a/dynarray.c b/dynarray.c
new file mode 100644
index 0000000..26a6058
--- /dev/null
+++ b/dynarray.c
@@ -0,0 +1,719 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include "foma.h"
+
+#define INITIAL_SIZE 16384
+#define SIGMA_HASH_SIZE 1021
+#define MINSIGMA 3
+
+struct foma_reserved_symbols {
+ char *symbol;
+ int number;
+ char *prints_as;
+} foma_reserved_symbols[] = {
+ {"@_EPSILON_SYMBOL_@" , EPSILON , "0"},
+ {"@_UNKNOWN_SYMBOL_@" , UNKNOWN , "?"},
+ {"@_IDENTITY_SYMBOL_@", IDENTITY, "@"},
+ {NULL,0,NULL}
+};
+
+static size_t current_fsm_size;
+static unsigned int current_fsm_linecount, current_state_no, current_final, current_start, current_trans, num_finals, num_initials, arity, statecount;
+static _Bool is_deterministic, is_epsilon_free;
+static struct fsm_state *current_fsm_head;
+
+static unsigned int mainloop, ssize, arccount;
+
+struct sigma_lookup {
+ int target;
+ unsigned int mainloop;
+};
+
+static struct sigma_lookup *slookup;
+
+/* Functions for directly building a fsm_state structure */
+/* dynamically. */
+
+/* fsm_state_init() is called when a new machine is constructed */
+
+/* fsm_state_add_arc() adds an arc and possibly reallocs the array */
+
+/* fsm_state_close() adds the sentinel entry and clears values */
+
+struct fsm_state *fsm_state_init(int sigma_size) {
+ current_fsm_head = xxmalloc(INITIAL_SIZE * sizeof(struct fsm_state));
+ current_fsm_size = INITIAL_SIZE;
+ current_fsm_linecount = 0;
+ ssize = sigma_size+1;
+ slookup = xxcalloc(ssize*ssize,sizeof(struct sigma_lookup));
+ mainloop = 1;
+ is_deterministic = 1;
+ is_epsilon_free = 1;
+ arccount = 0;
+ num_finals = 0;
+ num_initials = 0;
+ statecount = 0;
+ arity = 1;
+ current_trans = 1;
+ return(current_fsm_head);
+}
+
+void fsm_state_set_current_state(int state_no, int final_state, int start_state) {
+ current_state_no = state_no;
+ current_final = final_state;
+ current_start = start_state;
+ current_trans = 0;
+ if (current_final == 1)
+ num_finals++;
+ if (current_start == 1)
+ num_initials++;
+}
+
+/* Add sentinel if needed */
+void fsm_state_end_state() {
+ if (current_trans == 0) {
+ fsm_state_add_arc(current_state_no, -1, -1, -1, current_final, current_start);
+ }
+ statecount++;
+ mainloop++;
+}
+
+void fsm_state_add_arc(int state_no, int in, int out, int target, int final_state, int start_state) {
+ struct fsm_state *cptr;
+
+ if (in != out) {
+ arity = 2;
+ }
+ /* Check epsilon moves */
+ if (in == EPSILON && out == EPSILON) {
+ if (state_no == target) {
+ return;
+ } else {
+ is_deterministic = 0;
+ is_epsilon_free = 0;
+ }
+ }
+
+ /* Check if we already added this particular arc and skip */
+ /* Also check if net becomes non-det */
+ if (in != -1 && out != -1) {
+ if ((slookup+(ssize*in)+out)->mainloop == mainloop) {
+ if ((slookup+(ssize*in)+out)->target == target) {
+ return;
+ } else {
+ is_deterministic = 0;
+ }
+ }
+ arccount++;
+ (slookup+(ssize*in)+out)->mainloop = mainloop;
+ (slookup+(ssize*in)+out)->target = target;
+ }
+
+ current_trans = 1;
+ if (current_fsm_linecount >= current_fsm_size) {
+ current_fsm_size *= 2;
+ current_fsm_head = xxrealloc(current_fsm_head, current_fsm_size * sizeof(struct fsm_state));
+ if (current_fsm_head == NULL) {
+ perror("Fatal error: out of memory\n");
+ exit(1);
+ }
+ }
+ cptr = current_fsm_head + current_fsm_linecount;
+ cptr->state_no = state_no;
+ cptr->in = in;
+ cptr->out = out;
+ cptr->target = target;
+ cptr->final_state = final_state;
+ cptr->start_state = start_state;
+ current_fsm_linecount++;
+}
+
+void fsm_state_close(struct fsm *net) {
+ fsm_state_add_arc(-1,-1,-1,-1,-1,-1);
+ current_fsm_head = xxrealloc(current_fsm_head, current_fsm_linecount * sizeof(struct fsm_state));
+ net->arity = arity;
+ net->arccount = arccount;
+ net->statecount = statecount;
+ net->linecount = current_fsm_linecount;
+ net->finalcount = num_finals;
+ net->pathcount = PATHCOUNT_UNKNOWN;
+ if (num_initials > 1)
+ is_deterministic = 0;
+ net->is_deterministic = is_deterministic;
+ net->is_pruned = UNK;
+ net->is_minimized = UNK;
+ net->is_epsilon_free = is_epsilon_free;
+ net->is_loop_free = UNK;
+ net->is_completed = UNK;
+ net->arcs_sorted_in = 0;
+ net->arcs_sorted_out = 0;
+
+ net->states = current_fsm_head;
+ xxfree(slookup);
+}
+
+/* Construction functions */
+
+struct fsm_construct_handle *fsm_construct_init(char *name) {
+ struct fsm_construct_handle *handle;
+ handle = xxmalloc(sizeof(struct fsm_construct_handle));
+ handle->fsm_state_list = xxcalloc(1024,sizeof(struct fsm_state_list));
+ handle->fsm_state_list_size = 1024;
+ handle->fsm_sigma_list = xxcalloc(1024,sizeof(struct fsm_sigma_list));
+ handle->fsm_sigma_list_size = 1024;
+ handle->fsm_sigma_hash = xxcalloc(SIGMA_HASH_SIZE,sizeof(struct fsm_sigma_hash));
+ handle->maxstate = -1;
+ handle->maxsigma = -1;
+ handle->numfinals = 0;
+ if (name == NULL) {
+ handle->name = NULL;
+ } else {
+ handle->name = xxstrdup(name);
+ }
+ handle->hasinitial = 0;
+ return(handle);
+}
+
+void fsm_construct_check_size(struct fsm_construct_handle *handle, int state_no) {
+ int i, oldsize, newsize;
+ struct fsm_state_list *sl;
+ oldsize = handle->fsm_state_list_size;
+ if (oldsize <= state_no) {
+ newsize = next_power_of_two(state_no);
+ handle->fsm_state_list = xxrealloc(handle->fsm_state_list, newsize*sizeof(struct fsm_state_list));
+ handle->fsm_state_list_size = newsize;
+ sl = handle->fsm_state_list;
+ for (i=oldsize; i<newsize;i++) {
+ (sl+i)->is_final = 0;
+ (sl+i)->is_initial = 0;
+ (sl+i)->used = 0;
+ (sl+i)->num_trans = 0;
+ (sl+i)->fsm_trans_list = NULL;
+ }
+ }
+}
+
+void fsm_construct_set_final(struct fsm_construct_handle *handle, int state_no) {
+ struct fsm_state_list *sl;
+ fsm_construct_check_size(handle, state_no);
+
+ if (state_no > handle->maxstate)
+ handle->maxstate = state_no;
+
+ sl = handle->fsm_state_list;
+ if (!(sl+state_no)->is_final) {
+ (sl+state_no)->is_final = 1;
+ handle->numfinals++;
+ }
+}
+
+void fsm_construct_set_initial(struct fsm_construct_handle *handle, int state_no) {
+ struct fsm_state_list *sl;
+ fsm_construct_check_size(handle, state_no);
+
+ if (state_no > handle->maxstate)
+ handle->maxstate = state_no;
+
+ sl = handle->fsm_state_list;
+ (sl+state_no)->is_initial = 1;
+ handle->hasinitial = 1;
+}
+
+void fsm_construct_add_arc(struct fsm_construct_handle *handle, int source, int target, char *in, char *out) {
+ struct fsm_state_list *sl;
+ struct fsm_trans_list *tl;
+ int symin, symout;
+ fsm_construct_check_size(handle, source);
+ fsm_construct_check_size(handle, target);
+
+ if (source > handle->maxstate)
+ handle->maxstate = source;
+ if (target > handle->maxstate)
+ handle->maxstate = target;
+
+ sl = (handle->fsm_state_list)+target;
+ sl->used = 1;
+ sl = (handle->fsm_state_list)+source;
+ sl->used = 1;
+ tl = xxmalloc(sizeof(struct fsm_trans_list));
+ tl->next = sl->fsm_trans_list;
+ sl->fsm_trans_list = tl;
+ if ((symin = fsm_construct_check_symbol(handle,in)) == -1)
+ symin = fsm_construct_add_symbol(handle,in);
+ if ((symout = fsm_construct_check_symbol(handle,out)) == -1)
+ symout = fsm_construct_add_symbol(handle,out);
+ tl->in = symin;
+ tl->out = symout;
+ tl->target = target;
+}
+
+unsigned int fsm_construct_hash_sym(char *symbol) {
+ register unsigned int hash;
+ hash = 0;
+
+ while (*symbol != '\0')
+ hash = hash + *symbol++;
+ return (hash % SIGMA_HASH_SIZE);
+}
+
+void fsm_construct_add_arc_nums(struct fsm_construct_handle *handle, int source, int target, int in, int out) {
+ struct fsm_state_list *sl;
+ struct fsm_trans_list *tl;
+ fsm_construct_check_size(handle, source);
+ fsm_construct_check_size(handle, target);
+
+ if (source > handle->maxstate)
+ handle->maxstate = source;
+ if (target > handle->maxstate)
+ handle->maxstate = target;
+
+ sl = (handle->fsm_state_list)+target;
+ sl->used = 1;
+ sl = (handle->fsm_state_list)+source;
+ sl->used = 1;
+ tl = xxmalloc(sizeof(struct fsm_trans_list));
+ tl->next = sl->fsm_trans_list;
+ sl->fsm_trans_list = tl;
+ tl->in = in;
+ tl->out = out;
+ tl->target = target;
+}
+
+/* Copies entire alphabet from existing network */
+
+void fsm_construct_copy_sigma(struct fsm_construct_handle *handle, struct sigma *sigma) {
+
+ unsigned int hash;
+ int symnum;
+ struct fsm_sigma_hash *fh, *newfh;
+ char *symbol, *symdup;
+
+ for (; sigma != NULL && sigma->number != -1; sigma = sigma->next) {
+ symnum = sigma->number;
+ if (symnum > handle->maxsigma) {
+ handle->maxsigma = symnum;
+ }
+ symbol = sigma->symbol;
+ if (symnum >= handle->fsm_sigma_list_size) {
+ handle->fsm_sigma_list_size = next_power_of_two(handle->fsm_sigma_list_size);
+ handle->fsm_sigma_list = xxrealloc(handle->fsm_sigma_list, (handle->fsm_sigma_list_size) * sizeof(struct fsm_sigma_list));
+ }
+ /* Insert into list */
+ symdup = xxstrdup(symbol);
+ ((handle->fsm_sigma_list)+symnum)->symbol = symdup;
+
+ /* Insert into hashtable */
+ hash = fsm_construct_hash_sym(symbol);
+ fh = (handle->fsm_sigma_hash)+hash;
+ if (fh->symbol == NULL) {
+ fh->symbol = symdup;
+ fh->sym = symnum;
+ } else {
+ newfh = xxcalloc(1,sizeof(struct fsm_sigma_hash));
+ newfh->next = fh->next;
+ fh->next = newfh;
+ newfh->symbol = symdup;
+ newfh->sym = symnum;
+ }
+ }
+}
+
+int fsm_construct_add_symbol(struct fsm_construct_handle *handle, char *symbol) {
+ int i, symnum, reserved;
+ unsigned int hash;
+ struct fsm_sigma_hash *fh, *newfh;
+ char *symdup;
+
+ /* Is symbol reserved? */
+ for (i=0, reserved = 0; foma_reserved_symbols[i].symbol != NULL; i++) {
+ if (strcmp(symbol, foma_reserved_symbols[i].symbol) == 0) {
+ symnum = foma_reserved_symbols[i].number;
+ reserved = 1;
+ if (handle->maxsigma < symnum) {
+ handle->maxsigma = symnum;
+ }
+ break;
+ }
+ }
+
+ if (reserved == 0) {
+ symnum = handle->maxsigma + 1;
+ if (symnum < MINSIGMA)
+ symnum = MINSIGMA;
+ handle->maxsigma = symnum;
+ }
+
+ if (symnum >= handle->fsm_sigma_list_size) {
+ handle->fsm_sigma_list_size = next_power_of_two(handle->fsm_sigma_list_size);
+ handle->fsm_sigma_list = xxrealloc(handle->fsm_sigma_list, (handle->fsm_sigma_list_size) * sizeof(struct fsm_sigma_list));
+ }
+ /* Insert into list */
+ symdup = xxstrdup(symbol);
+ ((handle->fsm_sigma_list)+symnum)->symbol = symdup;
+
+ /* Insert into hashtable */
+ hash = fsm_construct_hash_sym(symbol);
+ fh = (handle->fsm_sigma_hash)+hash;
+ if (fh->symbol == NULL) {
+ fh->symbol = symdup;
+ fh->sym = symnum;
+ } else {
+ newfh = xxcalloc(1,sizeof(struct fsm_sigma_hash));
+ newfh->next = fh->next;
+ fh->next = newfh;
+ newfh->symbol = symdup;
+ newfh->sym = symnum;
+ }
+ return symnum;
+}
+
+int fsm_construct_check_symbol(struct fsm_construct_handle *handle, char *symbol) {
+ int hash;
+ struct fsm_sigma_hash *fh;
+ hash = fsm_construct_hash_sym(symbol);
+ fh = (handle->fsm_sigma_hash)+hash;
+ if (fh->symbol == NULL)
+ return -1;
+ for (; fh != NULL; fh = fh->next) {
+ if (strcmp(symbol,fh->symbol) == 0) {
+ return (fh->sym);
+ }
+ }
+ return -1;
+}
+
+struct sigma *fsm_construct_convert_sigma(struct fsm_construct_handle *handle) {
+ struct fsm_sigma_list *sl;
+ struct sigma *sigma, *oldsigma, *newsigma;
+ int i;
+ oldsigma = sigma = NULL;
+ sl = handle->fsm_sigma_list;
+ for (i=0; i <= handle->maxsigma; i++) {
+ if ((sl+i)->symbol != NULL) {
+ newsigma = xxmalloc(sizeof(struct sigma));
+ newsigma->number = i;
+ newsigma->symbol = (sl+i)->symbol;
+ newsigma->next = NULL;
+ if (oldsigma != NULL) {
+ oldsigma->next = newsigma;
+ } else {
+ sigma = newsigma;
+ }
+ oldsigma = newsigma;
+ }
+ }
+ return(sigma);
+}
+
+struct fsm *fsm_construct_done(struct fsm_construct_handle *handle) {
+ int i, emptyfsm;
+ struct fsm *net;
+ struct fsm_state_list *sl;
+ struct fsm_trans_list *trans, *transnext;
+ struct fsm_sigma_hash *sigmahash, *sigmahashnext;
+
+ sl = handle->fsm_state_list;
+ if (handle->maxstate == -1 || handle->numfinals == 0 || handle->hasinitial == 0) {
+ return(fsm_empty_set());
+ }
+ fsm_state_init((handle->maxsigma)+1);
+
+ for (i=0, emptyfsm = 1; i <= handle->maxstate; i++) {
+ fsm_state_set_current_state(i, (sl+i)->is_final, (sl+i)->is_initial);
+ if ((sl+i)->is_initial && (sl+i)->is_final)
+ emptyfsm = 0; /* We want to keep track of if FSM has (a) something outgoing from initial, or (b) initial is final */
+ for (trans = (sl+i)->fsm_trans_list; trans != NULL; trans = trans->next) {
+ if ((sl+i)->is_initial)
+ emptyfsm = 0;
+ fsm_state_add_arc(i, trans->in, trans->out, trans->target, (sl+i)->is_final, (sl+i)->is_initial);
+ }
+ fsm_state_end_state();
+ }
+ net = fsm_create("");
+ sprintf(net->name, "%X",rand());
+ xxfree(net->sigma);
+ fsm_state_close(net);
+
+ net->sigma = fsm_construct_convert_sigma(handle);
+ if (handle->name != NULL) {
+ strncpy(net->name, handle->name, 40);
+ xxfree(handle->name);
+ } else {
+ sprintf(net->name, "%X",rand());
+ }
+
+ /* Free transitions */
+ for (i=0; i < handle->fsm_state_list_size; i++) {
+ trans = (((handle->fsm_state_list)+i)->fsm_trans_list);
+ while (trans != NULL) {
+ transnext = trans->next;
+ xxfree(trans);
+ trans = transnext;
+ }
+ }
+ /* Free hash table */
+ for (i=0; i < SIGMA_HASH_SIZE; i++) {
+ sigmahash = (((handle->fsm_sigma_hash)+i)->next);
+ while (sigmahash != NULL) {
+ sigmahashnext = sigmahash->next;
+ xxfree(sigmahash);
+ sigmahash = sigmahashnext;
+ }
+ }
+ xxfree(handle->fsm_sigma_list);
+ xxfree(handle->fsm_sigma_hash);
+ xxfree(handle->fsm_state_list);
+ xxfree(handle);
+ sigma_sort(net);
+ if (emptyfsm) {
+ fsm_destroy(net);
+ return(fsm_empty_set());
+ }
+ return(net);
+}
+
+/* Reading functions */
+
+int fsm_read_is_final(struct fsm_read_handle *h, int state) {
+ return (*((h->lookuptable)+state) & 2);
+}
+
+int fsm_read_is_initial(struct fsm_read_handle *h, int state) {
+ return (*((h->lookuptable)+state) & 1);
+}
+
+struct fsm_read_handle *fsm_read_init(struct fsm *net) {
+ struct fsm_read_handle *handle;
+ struct fsm_state *fsm, **states_head;
+ int i, j, k, num_states, num_initials, num_finals, sno, *finals_head, *initials_head, laststate;
+
+ unsigned char *lookuptable;
+ if (net == NULL) {return (NULL);}
+
+ num_states = net->statecount;
+ lookuptable = xxcalloc(num_states, sizeof(unsigned char));
+
+ num_initials = num_finals = 0;
+
+ handle = xxcalloc(1,sizeof(struct fsm_read_handle));
+ states_head = xxcalloc(num_states+1,sizeof(struct fsm **));
+
+ laststate = -1;
+ for (i=0, fsm=net->states; (fsm+i)->state_no != -1; i++) {
+ sno = (fsm+i)->state_no;
+ if ((fsm+i)->start_state) {
+ if (!(*(lookuptable+sno) & 1)) {
+ *(lookuptable+sno) |= 1;
+ num_initials++;
+ }
+
+ }
+ if ((fsm+i)->final_state) {
+ if (!(*(lookuptable+sno) & 2)) {
+ *(lookuptable+sno) |= 2;
+ num_finals++;
+ }
+ }
+ if ((fsm+i)->in == UNKNOWN || (fsm+i)->out == UNKNOWN || (fsm+i)->in == IDENTITY || (fsm+i)->out == IDENTITY) {
+ handle->has_unknowns = 1;
+ }
+ if ((fsm+i)->state_no != laststate) {
+ *(states_head+(fsm+i)->state_no) = fsm+i;
+ }
+ laststate = (fsm+i)->state_no;
+ }
+
+ finals_head = xxcalloc(num_finals+1,sizeof(int));
+ initials_head = xxcalloc(num_initials+1,sizeof(int));
+
+
+ for (i=j=k=0; i < num_states; i++) {
+ if (*(lookuptable+i) & 1) {
+ *(initials_head+j) = i;
+ j++;
+ }
+ if (*(lookuptable+i) & 2) {
+ *(finals_head+k) = i;
+ k++;
+ }
+ }
+ *(initials_head+j) = -1;
+ *(finals_head+k) = -1;
+
+ handle->finals_head = finals_head;
+ handle->initials_head = initials_head;
+ handle->states_head = states_head;
+
+ handle->fsm_sigma_list = sigma_to_list(net->sigma);
+ handle->sigma_list_size = sigma_max(net->sigma)+1;
+ handle->arcs_head = fsm;
+ handle->lookuptable = lookuptable;
+ handle->net = net;
+ return(handle);
+}
+
+void fsm_read_reset(struct fsm_read_handle *handle) {
+ if (handle == NULL)
+ return;
+ handle->arcs_cursor = NULL;
+ handle->initials_cursor = NULL;
+ handle->finals_cursor = NULL;
+ handle->states_cursor = NULL;
+}
+
+int fsm_get_next_state_arc(struct fsm_read_handle *handle) {
+ handle->arcs_cursor++;
+ if ((handle->arcs_cursor->state_no != handle->current_state) || (handle->arcs_cursor->target == -1)) {
+ handle->arcs_cursor--;
+ return 0;
+ }
+ return 1;
+}
+
+int fsm_get_next_arc(struct fsm_read_handle *handle) {
+ if (handle->arcs_cursor == NULL) {
+ handle->arcs_cursor = handle->arcs_head;
+ while (handle->arcs_cursor->state_no != -1 && handle->arcs_cursor->target == -1) {
+ handle->arcs_cursor++;
+ }
+ if (handle->arcs_cursor->state_no == -1) {
+ return 0;
+ }
+ } else {
+ if (handle->arcs_cursor->state_no == -1) {
+ return 0;
+ }
+ do {
+ handle->arcs_cursor++;
+ } while (handle->arcs_cursor->state_no != -1 && handle->arcs_cursor->target == -1);
+ if (handle->arcs_cursor->state_no == -1) {
+ return 0;
+ }
+ }
+ return 1;
+}
+
+int fsm_get_arc_source(struct fsm_read_handle *handle) {
+ if (handle->arcs_cursor == NULL) { return -1;}
+ return(handle->arcs_cursor->state_no);
+}
+
+int fsm_get_arc_target(struct fsm_read_handle *handle) {
+ if (handle->arcs_cursor == NULL) { return -1;}
+ return(handle->arcs_cursor->target);
+}
+
+int fsm_get_symbol_number(struct fsm_read_handle *handle, char *symbol) {
+ int i;
+ for (i=0; i < handle->sigma_list_size; i++) {
+ if ((handle->fsm_sigma_list+i)->symbol == NULL)
+ continue;
+ if (strcmp(symbol, (handle->fsm_sigma_list+i)->symbol) == 0) {
+ return i;
+ }
+ }
+ return -1;
+}
+
+char *fsm_get_arc_in(struct fsm_read_handle *handle) {
+ int index;
+ char *sym;
+ if (handle->arcs_cursor == NULL) { return NULL;}
+ index = handle->arcs_cursor->in;
+ sym = (handle->fsm_sigma_list+index)->symbol;
+ return(sym);
+}
+
+int fsm_get_arc_num_in(struct fsm_read_handle *handle) {
+ if (handle->arcs_cursor == NULL) { return -1;}
+ return(handle->arcs_cursor->in);
+}
+
+int fsm_get_arc_num_out(struct fsm_read_handle *handle) {
+ if (handle->arcs_cursor == NULL) { return -1;}
+ return(handle->arcs_cursor->out);
+}
+
+char *fsm_get_arc_out(struct fsm_read_handle *handle) {
+ int index;
+ char *sym;
+ if (handle->arcs_cursor == NULL) { return NULL; }
+ index = handle->arcs_cursor->out;
+ sym = (handle->fsm_sigma_list+index)->symbol;
+ return(sym);
+}
+
+int fsm_get_next_initial(struct fsm_read_handle *handle) {
+ if (handle->initials_cursor == NULL) {
+ handle->initials_cursor = handle->initials_head;
+ } else {
+ if (*(handle->initials_cursor) == -1) {
+ return -1;
+ }
+ handle->initials_cursor++;
+ }
+ return *(handle->initials_cursor);
+}
+
+int fsm_get_next_final(struct fsm_read_handle *handle) {
+ if (handle->finals_cursor == NULL) {
+ handle->finals_cursor = handle->finals_head;
+ } else {
+ if (*(handle->finals_cursor) == -1) {
+ return -1;
+ }
+ handle->finals_cursor++;
+ }
+ return *(handle->finals_cursor);
+}
+
+int fsm_get_num_states(struct fsm_read_handle *handle) {
+ return(handle->net->statecount);
+}
+
+int fsm_get_has_unknowns(struct fsm_read_handle *handle) {
+ return(handle->has_unknowns);
+}
+
+int fsm_get_next_state(struct fsm_read_handle *handle) {
+ int stateno;
+ if (handle->states_cursor == NULL) {
+ handle->states_cursor = handle->states_head;
+ } else {
+ handle->states_cursor++;
+ }
+ if (handle->states_cursor - handle->states_head >= fsm_get_num_states(handle)) {
+ return -1;
+ }
+ handle->arcs_cursor = *(handle->states_cursor);
+ stateno = (*(handle->states_cursor))->state_no;
+ handle->arcs_cursor--;
+ handle->current_state = stateno;
+ return (stateno);
+}
+
+void fsm_read_done(struct fsm_read_handle *handle) {
+ xxfree(handle->lookuptable);
+ xxfree(handle->fsm_sigma_list);
+ xxfree(handle->finals_head);
+ xxfree(handle->initials_head);
+ xxfree(handle->states_head);
+ xxfree(handle);
+}
diff --git a/extract.c b/extract.c
new file mode 100644
index 0000000..7bc8259
--- /dev/null
+++ b/extract.c
@@ -0,0 +1,71 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include "foma.h"
+
+struct fsm *fsm_lower(struct fsm *net) {
+ struct fsm_state *fsm;
+ int i, prevstate, out;
+ fsm = net->states;
+ fsm_state_init(sigma_max(net->sigma));
+ prevstate = -1;
+ for (i = 0; (fsm+i)->state_no != - 1; prevstate = (fsm+i)->state_no, i++) {
+ if (prevstate != -1 && prevstate != (fsm+i)->state_no) {
+ fsm_state_end_state();
+ }
+ if (prevstate != (fsm+i)->state_no) {
+ fsm_state_set_current_state((fsm+i)->state_no, (fsm+i)->final_state, (fsm+i)->start_state);
+ }
+ if ((fsm+i)->target != -1) {
+ out = ((fsm+i)->out == UNKNOWN) ? IDENTITY : (fsm+i)->out;
+ fsm_state_add_arc((fsm+i)->state_no, out, out, (fsm+i)->target, (fsm+i)->final_state, (fsm+i)->start_state);
+ }
+ }
+ fsm_state_end_state();
+ xxfree(net->states);
+ fsm_state_close(net);
+ fsm_update_flags(net,NO,NO,NO,UNK,UNK,UNK);
+ sigma_cleanup(net,0);
+ return(net);
+}
+
+struct fsm *fsm_upper(struct fsm *net) {
+ struct fsm_state *fsm;
+ int i, prevstate, in;
+ fsm = net->states;
+ fsm_state_init(sigma_max(net->sigma));
+ prevstate = -1;
+ for (i = 0; (fsm+i)->state_no != - 1; prevstate = (fsm+i)->state_no, i++) {
+ if (prevstate != -1 && prevstate != (fsm+i)->state_no) {
+ fsm_state_end_state();
+ }
+ if (prevstate != (fsm+i)->state_no) {
+ fsm_state_set_current_state((fsm+i)->state_no, (fsm+i)->final_state, (fsm+i)->start_state);
+ }
+ if ((fsm+i)->target != -1) {
+ in = ((fsm+i)->in == UNKNOWN) ? IDENTITY : (fsm+i)->in;
+ fsm_state_add_arc((fsm+i)->state_no, in, in, (fsm+i)->target, (fsm+i)->final_state, (fsm+i)->start_state);
+ }
+ }
+ fsm_state_end_state();
+ xxfree(net->states);
+ fsm_state_close(net);
+ fsm_update_flags(net,NO,NO,NO,UNK,UNK,UNK);
+ sigma_cleanup(net,0);
+ return(net);
+}
diff --git a/flags.c b/flags.c
new file mode 100644
index 0000000..b7ac3d1
--- /dev/null
+++ b/flags.c
@@ -0,0 +1,518 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <locale.h>
+#include "foma.h"
+
+#define FAIL 1
+#define SUCCEED 2
+#define NONE 3
+
+static struct flags *flag_extract (struct fsm *net);
+static char *flag_type_to_char (int type);
+static void flag_purge (struct fsm *net, char *name);
+static struct fsm *flag_create_symbol(int type, char *name, char *value);
+
+struct flags {
+ int type;
+ char *name;
+ char *value;
+ struct flags *next;
+};
+
+/* We eliminate all flags by creating a list of them and building a regex filter */
+/* that successively removes unwanted paths. NB: flag_eliminate() called with the */
+/* second argument NULL eliminates all flags. */
+/* The regexes we build for each flag symbol are of the format: */
+/* ~[?* FAIL ~$SUCCEED THISFLAG ?*] for U,P,D */
+/* or */
+/* ~[(?* FAIL) ~$SUCCEED THISFLAG ?*] for the R flag */
+/* The function flag_build() determines, depending on the flag at hand for each */
+/* of the other flags occurring in the network if it belongs in FAIL, SUCCEED, */
+/* or neither. */
+/* The languages FAIL, SUCCEED is then the union of all symbols that cause */
+/* compatibility or incompatibility. */
+/* We intersect all these filters, creating a large filter that we compose both on */
+/* the upper side of the network and the lower side: */
+/* RESULT = FILTER .o. ORIGINAL .o. FILTER */
+/* We can't simply intersect the language with FILTER because the lower side flags */
+/* are independent of the upper side ones, and the network may be a transducer. */
+/* Finally, we replace the affected arcs with EPSILON arcs, and call */
+/* sigma_cleanup() to purge the symbols not occurring on arcs. */
+
+///
+///Eliminate a flag from a network. If called with name = NULL, eliminate all flags.
+///
+
+struct fsm *flag_eliminate(struct fsm *net, char *name) {
+
+ struct flags *flags, *f, *ff;
+ struct fsm *filter, *succeed_flags, *fail_flags, *self, *newfilter, *newnet;
+ int flag, fstatus, found;
+
+ filter = NULL;
+
+ flags = flag_extract(net);
+ /* Check that flag actually exists in net */
+ if (name != NULL) {
+ for (found = 0, f = flags; f != NULL; f = f->next) {
+ if (strcmp(name,f->name) == 0)
+ found = 1;
+ }
+ if (found == 0) {
+ fprintf(stderr,"Flag attribute '%s' does not occur in the network.\n",name);
+ return(net);
+ }
+ }
+
+ flag = 0;
+
+ for (f = flags; f != NULL; f = f->next) {
+
+ if ((name == NULL || strcmp(f->name,name) == 0) &&
+ (f->type | FLAG_UNIFY | FLAG_REQUIRE | FLAG_DISALLOW | FLAG_EQUAL)) {
+
+ succeed_flags = fsm_empty_set();
+ fail_flags = fsm_empty_set();
+ self = flag_create_symbol(f->type, f->name, f->value);
+
+ for (ff = flags, flag = 0; ff != NULL; ff = ff->next) {
+ fstatus = flag_build(f->type, f->name, f->value, ff->type, ff->name, ff->value);
+ if (fstatus == FAIL) {
+ fail_flags = fsm_minimize(fsm_union(fail_flags, flag_create_symbol(ff->type, ff->name, ff->value)));
+ flag = 1;
+ }
+ if (fstatus == SUCCEED) {
+ succeed_flags = fsm_minimize(fsm_union(succeed_flags, flag_create_symbol(ff->type, ff->name, ff->value)));
+ flag = 1;
+ }
+ }
+ }
+
+ if (flag) {
+ if (f->type == FLAG_REQUIRE) {
+ newfilter = fsm_complement(fsm_concat(fsm_optionality(fsm_concat(fsm_universal(), fail_flags)), fsm_concat(fsm_complement(fsm_contains(succeed_flags)), fsm_concat(self, fsm_universal()))));
+
+ } else {
+ newfilter = fsm_complement(fsm_contains(fsm_concat(fail_flags,fsm_concat(fsm_complement(fsm_contains(succeed_flags)),self))));
+ }
+
+ filter = (filter == NULL) ? newfilter : fsm_intersect(filter, newfilter);
+ }
+ flag = 0;
+ }
+ if (filter != NULL) {
+ extern int g_flag_is_epsilon;
+ int old_g_flag_is_epsilon;
+ old_g_flag_is_epsilon = g_flag_is_epsilon;
+ g_flag_is_epsilon = 0;
+ newnet = fsm_compose(fsm_copy(filter),fsm_compose(net,fsm_copy(filter)));
+ g_flag_is_epsilon = old_g_flag_is_epsilon;
+ } else {
+ newnet = net;
+ }
+ flag_purge(newnet, name);
+ newnet = fsm_minimize(newnet);
+ sigma_cleanup(newnet,0);
+ xxfree(flags);
+ return(fsm_topsort(newnet));
+}
+
+struct fsm *flag_create_symbol(int type, char *name, char *value) {
+ char *string;
+ if (value == NULL)
+ value = "";
+
+ string = xxmalloc(sizeof(char)*strlen(name)+strlen(value)+6);
+ *string = '\0';
+ strcat(string, "@");
+ strcat(string, flag_type_to_char(type));
+ strcat(string, ".");
+ strcat(string, name);
+ if (strcmp(value,"") != 0) {
+ strcat(string, ".");
+ strcat(string, value);
+ }
+ strcat(string, "@");
+
+ return(fsm_symbol(string));
+
+}
+
+char *flag_type_to_char (int type) {
+ switch(type) {
+ case FLAG_UNIFY:
+ return("U");
+ case FLAG_CLEAR:
+ return("C");
+ case FLAG_DISALLOW:
+ return("D");
+ case FLAG_NEGATIVE:
+ return("N");
+ case FLAG_POSITIVE:
+ return("P");
+ case FLAG_REQUIRE:
+ return("R");
+ case FLAG_EQUAL:
+ return("E");
+ }
+ return NULL;
+}
+
+int flag_build(int ftype, char *fname, char *fvalue, int fftype, char *ffname, char *ffvalue) {
+ int valeq, selfnull;
+
+ selfnull = 0; /* If current flag has no value, e.g. @R.A@ */
+ if (strcmp(fname,ffname) != 0)
+ return NONE;
+
+ if (fvalue == NULL) {
+ fvalue = "";
+ selfnull = 1;
+ }
+
+ if (ffvalue == NULL)
+ ffvalue = "";
+
+ valeq = strcmp(fvalue, ffvalue);
+ /* U flags */
+ if (ftype == FLAG_UNIFY && fftype == FLAG_POSITIVE && valeq == 0)
+ return SUCCEED;
+ if (ftype == FLAG_UNIFY && fftype == FLAG_CLEAR)
+ return SUCCEED;
+ if (ftype == FLAG_UNIFY && fftype == FLAG_UNIFY && valeq != 0)
+ return FAIL;
+ if (ftype == FLAG_UNIFY && fftype == FLAG_POSITIVE && valeq != 0)
+ return FAIL;
+ if (ftype == FLAG_UNIFY && fftype == FLAG_NEGATIVE && valeq == 0)
+ return FAIL;
+
+ /* R flag with value = 0 */
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_UNIFY && selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_POSITIVE && selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_NEGATIVE && selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_CLEAR && selfnull)
+ return FAIL;
+
+ /* R flag with value */
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_POSITIVE && valeq == 0 && !selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_UNIFY && valeq == 0 && !selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_POSITIVE && valeq != 0 && !selfnull)
+ return FAIL;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_UNIFY && valeq != 0 && !selfnull)
+ return FAIL;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_NEGATIVE && !selfnull)
+ return FAIL;
+ if (ftype == FLAG_REQUIRE && fftype == FLAG_CLEAR && !selfnull)
+ return FAIL;
+
+ /* D flag with value = 0 */
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_CLEAR && selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_POSITIVE && selfnull)
+ return FAIL;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_UNIFY && selfnull)
+ return FAIL;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_NEGATIVE && selfnull)
+ return FAIL;
+
+ /* D flag with value */
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_POSITIVE && valeq != 0 && !selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_CLEAR && !selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_NEGATIVE && valeq == 0 && !selfnull)
+ return SUCCEED;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_POSITIVE && valeq == 0 && !selfnull)
+ return FAIL;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_UNIFY && valeq == 0 && !selfnull)
+ return FAIL;
+ if (ftype == FLAG_DISALLOW && fftype == FLAG_NEGATIVE && valeq != 0 && !selfnull)
+ return FAIL;
+
+ return NONE;
+}
+
+
+/* Remove flags that are being eliminated from arcs and sigma */
+
+void flag_purge (struct fsm *net, char *name) {
+ struct fsm_state *fsm;
+ struct sigma *sigma;
+ int i, *ftable, sigmasize;
+ char *csym;
+ sigmasize = sigma_max(net->sigma)+1;
+ ftable = xxmalloc(sizeof(int) * sigmasize);
+ fsm = net->states;
+ for (i=0; i<sigmasize; i++)
+ *(ftable+i)=0;
+
+ for (sigma = net->sigma; sigma != NULL && sigma->number != -1; sigma = sigma->next) {
+
+ if (flag_check(sigma->symbol)) {
+ if (name == NULL) {
+ *(ftable+(sigma->number)) = 1;
+ } else {
+ csym = (sigma->symbol) + 3;
+ if (strncmp(csym,name,strlen(name)) == 0 && (strlen(csym)>strlen(name)) && (strncmp(csym+strlen(name),".",1) == 0 || strncmp(csym+strlen(name),"@",1) == 0)) {
+ *(ftable+(sigma->number)) = 1;
+ }
+ }
+ }
+ }
+ for (i = 0; i < sigmasize; i++) {
+ if (*(ftable+i)) {
+ net->sigma = sigma_remove_num(i, net->sigma);
+ }
+ }
+
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->in >= 0 && (fsm+i)->out >= 0) {
+ if (*(ftable+(fsm+i)->in))
+ (fsm+i)->in = EPSILON;
+ if (*(ftable+(fsm+i)->out))
+ (fsm+i)->out = EPSILON;
+ }
+ }
+
+ xxfree(ftable);
+ net->is_deterministic = net->is_minimized = net->is_epsilon_free = NO;
+ return;
+}
+
+/* Extract all flags from network and place them in struct flag linked list */
+
+struct flags *flag_extract (struct fsm *net) {
+ struct sigma *sigma;
+ struct flags *flags, *flagst;
+
+ flags = NULL;
+ for (sigma = net->sigma ; sigma != NULL; sigma = sigma->next) {
+ if (flag_check(sigma->symbol)) {
+ flagst = xxmalloc(sizeof(struct flags));
+ flagst->next = flags;
+ flags = flagst;
+
+ flags->type = flag_get_type(sigma->symbol);
+ flags->name = flag_get_name(sigma->symbol);
+ flags->value = flag_get_value(sigma->symbol);
+ }
+ }
+ return(flags);
+}
+
+int flag_check(char *s) {
+
+ /* We simply simulate this regex (where ND is not dot) */
+ /* "@" [U|P|N|R|E|D] "." ND+ "." ND+ "@" | "@" [D|R|C] "." ND+ "@" */
+ /* and return 1 if it matches */
+
+ int i;
+ i = 0;
+
+ if (*(s+i) == '@') { i++; goto s1; } return 0;
+ s1:
+ if (*(s+i) == 'C') { i++; goto s4; }
+ if (*(s+i) == 'N' || *(s+i) == 'E' || *(s+i) == 'U' || *(s+i) == 'P') { i++; goto s2; }
+ if (*(s+i) == 'R' || *(s+i) == 'D') { i++; goto s3; } return 0;
+ s2:
+ if (*(s+i) == '.') { i++; goto s5; } return 0;
+ s3:
+ if (*(s+i) == '.') { i++; goto s6; } return 0;
+ s4:
+ if (*(s+i) == '.') { i++; goto s7; } return 0;
+ s5:
+ if (*(s+i) != '.' && *(s+i) != '\0') { i++; goto s8; } return 0;
+ s6:
+ if (*(s+i) != '.' && *(s+i) != '\0') { i++; goto s9; } return 0;
+ s7:
+ if (*(s+i) != '.' && *(s+i) != '\0') { i++; goto s10; } return 0;
+ s8:
+ if (*(s+i) == '.') { i++; goto s7; }
+ if (*(s+i) != '.' && *(s+i) != '\0') { i++; goto s8; } return 0;
+ s9:
+ if (*(s+i) == '@') { i++; goto s11; }
+ if (*(s+i) == '.') { i++; goto s7; }
+ if (*(s+i) != '.' && *(s+i) != '\0') { i++; goto s9; } return 0;
+
+ s10:
+ if (*(s+i) == '@') {i++; goto s11;}
+ if (*(s+i) != '.' && *(s+i) != '\0') { i++; goto s10; } return 0;
+ s11:
+ if (*(s+i) == '\0') {return 1;} return 0;
+}
+
+int flag_get_type(char *string) {
+ if (strncmp(string+1,"U.",2) == 0) {
+ return FLAG_UNIFY;
+ }
+ if (strncmp(string+1,"C.",2) == 0) {
+ return FLAG_CLEAR;
+ }
+ if (strncmp(string+1,"D.",2) == 0) {
+ return FLAG_DISALLOW;
+ }
+ if (strncmp(string+1,"N.",2) == 0) {
+ return FLAG_NEGATIVE;
+ }
+ if (strncmp(string+1,"P.",2) == 0) {
+ return FLAG_POSITIVE;
+ }
+ if (strncmp(string+1,"R.",2) == 0) {
+ return FLAG_REQUIRE;
+ }
+ if (strncmp(string+1,"E.",2) == 0) {
+ return FLAG_EQUAL;
+ }
+ return 0;
+}
+
+char *flag_get_name(char *string) {
+ int i, start, end, len;
+ start = end = 0;
+ len = strlen(string);
+
+ for (i=0; i < len; i += (utf8skip(string+i) + 1)) {
+ if (*(string+i) == '.' && start == 0) {
+ start = i+1;
+ continue;
+ }
+ if ((*(string+i) == '.' || *(string+i) == '@') && start != 0) {
+ end = i;
+ break;
+ }
+ }
+ if (start > 0 && end > 0) {
+ return(xxstrndup(string+start,end-start));
+ }
+ return NULL;
+}
+
+char *flag_get_value(char *string) {
+ int i, first, start, end, len;
+ first = start = end = 0;
+ len = strlen(string);
+
+ for (i=0; i < len; i += (utf8skip(string+i) + 1)) {
+ if (*(string+i) == '.' && first == 0) {
+ first = i+1;
+ continue;
+ }
+ if (*(string+i) == '@' && start != 0) {
+ end = i;
+ break;
+ }
+ if (*(string+i) == '.' && first != 0) {
+ start = i+1;
+ continue;
+ }
+ }
+ if (start > 0 && end > 0) {
+ return(xxstrndup(string+start,end-start));
+ }
+ return NULL;
+}
+
+struct fsm *flag_twosided(struct fsm *net) {
+ struct fsm_state *fsm;
+ struct sigma *sigma;
+ int i, j, tail, *isflag, maxsigma, maxstate, newarcs, change;
+
+ /* Enforces twosided flag diacritics */
+
+ /* Mark flag symbols */
+ maxsigma = sigma_max(net->sigma);
+ isflag = xxcalloc(maxsigma+1, sizeof(int));
+ fsm = net->states;
+ for (sigma = net->sigma ; sigma != NULL; sigma = sigma->next) {
+ if (flag_check(sigma->symbol)) {
+ *(isflag+sigma->number) = 1;
+ } else {
+ *(isflag+sigma->number) = 0;
+ }
+ }
+ maxstate = 0;
+ change = 0;
+ for (i = 0, newarcs = 0; (fsm+i)->state_no != -1 ; i++) {
+ maxstate = (fsm+i)->state_no > maxstate ? (fsm+i)->state_no : maxstate;
+ if ((fsm+i)->target == -1)
+ continue;
+ if (*(isflag+(fsm+i)->in) && (fsm+i)->out == EPSILON) {
+ change = 1;
+ (fsm+i)->out = (fsm+i)->in;
+ }
+ else if (*(isflag+(fsm+i)->out) && (fsm+i)->in == EPSILON) {
+ change = 1;
+ (fsm+i)->in = (fsm+i)->out;
+ }
+ if ((*(isflag+(fsm+i)->in) || *(isflag+(fsm+i)->out)) && (fsm+i)->in != (fsm+i)->out) {
+ newarcs++;
+ }
+ }
+
+ if (newarcs == 0) {
+ if (change == 1) {
+ net->is_deterministic = UNK;
+ net->is_minimized = UNK;
+ net->is_pruned = UNK;
+ return fsm_topsort(fsm_minimize(net));
+ }
+ return net;
+ }
+ net->states = xxrealloc(net->states, sizeof(struct fsm)*(i+newarcs));
+ fsm = net->states;
+ tail = j = i;
+ maxstate++;
+ for (i = 0; i < tail; i++) {
+
+ if ((fsm+i)->target == -1)
+ continue;
+ if ((*(isflag+(fsm+i)->in) || *(isflag+(fsm+i)->out)) && (fsm+i)->in != (fsm+i)->out) {
+ if (*(isflag+(fsm+i)->in) && !*(isflag+(fsm+i)->out)) {
+ j = add_fsm_arc(fsm, j, maxstate, EPSILON, (fsm+i)->out, (fsm+i)->target, 0, 0);
+ (fsm+i)->out = (fsm+i)->in;
+ (fsm+i)->target = maxstate;
+ maxstate++;
+ }
+ else if (*(isflag+(fsm+i)->out) && !*(isflag+(fsm+i)->in)) {
+ j = add_fsm_arc(fsm, j, maxstate, (fsm+i)->out, (fsm+i)->out, (fsm+i)->target, 0, 0);
+ (fsm+i)->out = EPSILON;
+ (fsm+i)->target = maxstate;
+ maxstate++;
+ }
+ else if (*(isflag+(fsm+i)->in) && *(isflag+(fsm+i)->out)) {
+ j = add_fsm_arc(fsm, j, maxstate, (fsm+i)->out, (fsm+i)->out, (fsm+i)->target, 0, 0);
+ (fsm+i)->out = (fsm+i)->in;
+ (fsm+i)->target = maxstate;
+ maxstate++;
+ }
+ }
+ }
+ /* Add sentinel */
+ add_fsm_arc(fsm, j, -1, -1, -1, -1, -1, -1);
+ net->is_deterministic = UNK;
+ net->is_minimized = UNK;
+ return fsm_topsort(fsm_minimize(net));
+}
diff --git a/flookup.c b/flookup.c
new file mode 100644
index 0000000..05e5936
--- /dev/null
+++ b/flookup.c
@@ -0,0 +1,385 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include <ctype.h>
+#include <stdio.h>
+#include <limits.h>
+#include <getopt.h>
+#include <unistd.h>
+#include <sys/types.h>
+#include <sys/socket.h>
+#include <arpa/inet.h>
+#include "fomalib.h"
+
+#define LINE_LIMIT 262144
+#define UDP_MAX 65535
+#define FLOOKUP_PORT 6062
+
+static char *usagestring = "Usage: flookup [-h] [-a] [-i] [-s \"separator\"] [-w \"wordseparator\"] [-v] [-x] [-b] [-I <#|#k|#m|f>] [-S] [-P] [-A] <binary foma file>\n";
+
+static char *helpstring =
+"Applies words from stdin to a foma transducer/automaton read from a file and prints results to stdout.\n"
+
+"If the file contains several nets, inputs will be passed through all of them (simulating composition) or applied as alternates if the -a flag is specified (simulating priority union: the first net is tried first, if that fails to produce an output, then the second is tried, etc.).\n\n"
+"Options:\n\n"
+"-h\t\tprint help\n"
+"-a\t\ttry alternatives (in order of nets loaded, default is to pass words through each)\n"
+"-b\t\tunbuffered output (flushes output after each input word, for use in bidirectional piping)\n"
+"-i\t\tinverse application (apply down instead of up)\n"
+"-I indextype\tindex arcs with indextype (one of -I f -I #k -I #m or -I #)\n"
+"\t\t(usually slower than the default except for states > 1,000 arcs)\n"
+"\t\t -I # will index all states containing # arcs or more\n"
+"\t\t -I NUMk will index states from densest to sparsest until reaching mem limit of # kB\n"
+"\t\t -I NUMM will index states from densest to sparsest until reaching mem limit of # MB\n"
+"\t\t -I f will index flag-containing states only\n"
+"-q\t\tdon't sort arcs before applying (usually slower, except for really small, sparse automata)\n"
+"-S\t\trun flookup as UDP server (default addr INADDR_ANY port 6062)\n"
+"-A\t\t specify address of server\n"
+"-P\t\t specify port of server (default 6062)\n"
+"-s \"separator\"\tchange input/output separator symbol (default is TAB)\n"
+"-w \"separator\"\tchange words separator symbol (default is LF)\n"
+"-v\t\tprint version number\n"
+"-x\t\tdon't echo input string";
+
+struct lookup_chain {
+ struct fsm *net;
+ struct apply_handle *ah;
+ struct lookup_chain *next;
+ struct lookup_chain *prev;
+};
+
+#define DIR_DOWN 0
+#define DIR_UP 1
+
+static struct sockaddr_in serveraddr, clientaddr;
+static int listen_sd, numbytes;
+static socklen_t addrlen;
+
+static char buffer[2048];
+static int echo = 1, apply_alternates = 0, numnets = 0, direction = DIR_UP, results, buffered_output = 1, index_arcs = 0, index_flag_states = 0, index_cutoff = 0, index_mem_limit = INT_MAX, mode_server = 0, port_number = FLOOKUP_PORT, udpsize;
+static char *separator = "\t", *wordseparator = "\n", *server_address = NULL, *line, *serverstring = NULL;
+static FILE *INFILE;
+static struct lookup_chain *chain_head, *chain_tail, *chain_new, *chain_pos;
+static fsm_read_binary_handle fsrh;
+
+static char *(*applyer)() = &apply_up; /* Default apply direction = up */
+static void handle_line(char *s);
+static void app_print(char *result);
+static char *get_next_line();
+static void server_init();
+
+void app_print(char *result) {
+
+ if (!mode_server) {
+ if (echo == 1) {
+ fprintf(stdout, "%s%s",line, separator);
+ }
+ if (result == NULL) {
+ fprintf(stdout,"+?\n");
+ } else {
+ fprintf(stdout, "%s\n", result);
+ }
+ } else {
+ if (echo == 1) {
+ strncat(serverstring+udpsize, line, UDP_MAX-udpsize);
+ udpsize += strlen(line);
+ strncat(serverstring+udpsize, separator, UDP_MAX-udpsize);
+ udpsize += strlen(separator);
+ }
+ if (result == NULL) {
+ strncat(serverstring+udpsize, "?+\n", UDP_MAX-udpsize);
+ udpsize += 3;
+ } else {
+ strncat(serverstring+udpsize, result, UDP_MAX-udpsize);
+ udpsize += strlen(result);
+ strncat(serverstring+udpsize, "\n", UDP_MAX-udpsize);
+ udpsize++;
+ }
+ }
+}
+
+int main(int argc, char *argv[]) {
+ int opt, sortarcs = 1;
+ char *infilename;
+ struct fsm *net;
+
+ setvbuf(stdout, buffer, _IOFBF, sizeof(buffer));
+
+ while ((opt = getopt(argc, argv, "abhHiI:qs:SA:P:w:vx")) != -1) {
+ switch(opt) {
+ case 'a':
+ apply_alternates = 1;
+ break;
+ case 'b':
+ buffered_output = 0;
+ break;
+ case 'h':
+ printf("%s%s\n", usagestring,helpstring);
+ exit(0);
+ case 'i':
+ direction = DIR_DOWN;
+ applyer = &apply_down;
+ break;
+ case 'q':
+ sortarcs = 0;
+ break;
+ case 'I':
+ if (strcmp(optarg, "f") == 0) {
+ index_flag_states = 1;
+ index_arcs = 1;
+ } else if (strstr(optarg, "k") != NULL && strstr(optarg,"K") != NULL) {
+ /* k limit */
+ index_mem_limit = 1024*atoi(optarg);
+ index_arcs = 1;
+ } else if (strstr(optarg, "m") != NULL && strstr(optarg,"M") != NULL) {
+ /* m limit */
+ index_mem_limit = 1024*1024*atoi(optarg);
+ index_arcs = 1;
+ } else if (isdigit(*optarg)) {
+ index_arcs = 1;
+ index_cutoff = atoi(optarg);
+ }
+ break;
+ case 's':
+ separator = strdup(optarg);
+ break;
+ case 'S':
+ mode_server = 1;
+ break;
+ case 'A':
+ server_address = strdup(optarg);
+ break;
+ case 'P':
+ port_number = atoi(optarg);
+ break;
+ case 'w':
+ wordseparator = strdup(optarg);
+ break;
+ case 'v':
+ printf("flookup 1.03 (foma library version %s)\n", fsm_get_library_version_string());
+ exit(0);
+ case 'x':
+ echo = 0;
+ break;
+ default:
+ fprintf(stderr, "%s", usagestring);
+ exit(EXIT_FAILURE);
+ }
+ }
+ if (optind == argc) {
+ fprintf(stderr, "%s", usagestring);
+ exit(EXIT_FAILURE);
+ }
+
+ infilename = argv[optind];
+
+ if ((fsrh = fsm_read_binary_file_multiple_init(infilename)) == NULL) {
+ perror("File error");
+ exit(EXIT_FAILURE);
+ }
+ chain_head = chain_tail = NULL;
+
+ while ((net = fsm_read_binary_file_multiple(fsrh)) != NULL) {
+ numnets++;
+ chain_new = xxmalloc(sizeof(struct lookup_chain));
+ if (direction == DIR_DOWN && net->arcs_sorted_in != 1 && sortarcs) {
+ fsm_sort_arcs(net, 1);
+ }
+ if (direction == DIR_UP && net->arcs_sorted_out != 1 && sortarcs) {
+ fsm_sort_arcs(net, 2);
+ }
+ chain_new->net = net;
+ chain_new->ah = apply_init(net);
+ if (direction == DIR_DOWN && index_arcs) {
+ apply_index(chain_new->ah, APPLY_INDEX_INPUT, index_cutoff, index_mem_limit, index_flag_states);
+ }
+ if (direction == DIR_UP && index_arcs) {
+ apply_index(chain_new->ah, APPLY_INDEX_OUTPUT, index_cutoff, index_mem_limit, index_flag_states);
+ }
+
+ chain_new->next = NULL;
+ chain_new->prev = NULL;
+ if (chain_tail == NULL) {
+ chain_tail = chain_head = chain_new;
+ } else if (direction == DIR_DOWN || apply_alternates == 1) {
+ chain_tail->next = chain_new;
+ chain_new->prev = chain_tail;
+ chain_tail = chain_new;
+ } else {
+ chain_new->next = chain_head;
+ chain_head->prev = chain_new;
+ chain_head = chain_new;
+ }
+ }
+
+ if (numnets < 1) {
+ fprintf(stderr, "%s: %s\n", "File error", infilename);
+ exit(EXIT_FAILURE);
+ }
+
+ if (mode_server) {
+ server_init();
+ serverstring = xxcalloc(UDP_MAX+1, sizeof(char));
+ line = xxcalloc(UDP_MAX+1, sizeof(char));
+ addrlen = sizeof(clientaddr);
+ for (;;) {
+ numbytes = recvfrom(listen_sd, line, UDP_MAX, 0,(struct sockaddr *)&clientaddr, &addrlen);
+ if (numbytes == -1) {
+ perror("recvfrom() failed, aborting");
+ break;
+ }
+ line[numbytes] = '\0';
+ line[strcspn(line, "\n\r")] = '\0';
+ fflush(stdout);
+ results = 0;
+ udpsize = 0;
+ serverstring[0] = '\0';
+ handle_line(line);
+ if (results == 0) {
+ app_print(NULL);
+ }
+ if (serverstring[0] != '\0') {
+ numbytes = sendto(listen_sd, serverstring, strlen(serverstring), 0, (struct sockaddr *)&clientaddr, addrlen);
+ if (numbytes < 0) {
+ perror("sendto() failed"); fflush(stdout);
+ }
+ }
+ }
+ } else {
+ /* Standard read from stdin */
+ line = xxcalloc(LINE_LIMIT, sizeof(char));
+ INFILE = stdin;
+ while (get_next_line() != NULL) {
+ results = 0;
+ handle_line(line);
+ if (results == 0) {
+ app_print(NULL);
+ }
+ fprintf(stdout, "%s", wordseparator);
+ if (!buffered_output) {
+ fflush(stdout);
+ }
+ }
+ }
+ /* Cleanup */
+ for (chain_pos = chain_head; chain_pos != NULL; chain_pos = chain_head) {
+ chain_head = chain_pos->next;
+ if (chain_pos->ah != NULL) {
+ apply_clear(chain_pos->ah);
+ }
+ if (chain_pos->net != NULL) {
+ fsm_destroy(chain_pos->net);
+ }
+ xxfree(chain_pos);
+ }
+ if (serverstring != NULL)
+ xxfree(serverstring);
+ if (line != NULL)
+ xxfree(line);
+ exit(0);
+}
+
+char *get_next_line() {
+ char *r;
+ if ((r = fgets(line, LINE_LIMIT, INFILE)) != NULL) {
+ line[strcspn(line, "\n\r")] = '\0';
+ }
+ return r;
+}
+
+void handle_line(char *s) {
+ char *result, *tempstr;
+ /* Apply alternative */
+ if (apply_alternates == 1) {
+ for (chain_pos = chain_head, tempstr = s; ; chain_pos = chain_pos->next) {
+ result = applyer(chain_pos->ah, tempstr);
+ if (result != NULL) {
+ results++;
+ app_print(result);
+ while ((result = applyer(chain_pos->ah, NULL)) != NULL) {
+ results++;
+ app_print(result);
+ }
+ break;
+ }
+ if (chain_pos == chain_tail) {
+ break;
+ }
+ }
+ } else {
+
+ /* Get result from chain */
+ for (chain_pos = chain_head, tempstr = s; ; chain_pos = chain_pos->next) {
+ result = applyer(chain_pos->ah, tempstr);
+ if (result != NULL && chain_pos != chain_tail) {
+ tempstr = result;
+ continue;
+ }
+ if (result != NULL && chain_pos == chain_tail) {
+ do {
+ results++;
+ app_print(result);
+ } while ((result = applyer(chain_pos->ah, NULL)) != NULL);
+ }
+ if (result == NULL) {
+ /* Move up */
+ for (chain_pos = chain_pos->prev; chain_pos != NULL; chain_pos = chain_pos->prev) {
+ result = applyer(chain_pos->ah, NULL);
+ if (result != NULL) {
+ tempstr = result;
+ break;
+ }
+ }
+ }
+ if (chain_pos == NULL) {
+ break;
+ }
+ }
+ }
+}
+
+void server_init(void) {
+ unsigned int rcvsize = 262144;
+
+ if ((listen_sd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)) == -1) {
+ perror("socket() failed");
+ exit(1);
+ }
+ if (setsockopt(listen_sd, SOL_SOCKET, SO_RCVBUF, (char *) &rcvsize, sizeof(rcvsize)) < 0) {
+ perror("setsockopt() failed");
+ exit(1);
+ }
+ if (setsockopt(listen_sd, SOL_SOCKET, SO_SNDBUF, (char *) &rcvsize, sizeof(rcvsize)) < 0) {
+ perror("setsockopt() failed");
+ exit(1);
+ }
+
+ memset((char *) &serveraddr, 0, sizeof(serveraddr));
+ serveraddr.sin_family = AF_INET;
+ serveraddr.sin_port = htons(port_number);
+ if (server_address != NULL) {
+ serveraddr.sin_addr.s_addr = inet_addr(server_address);
+ } else {
+ serveraddr.sin_addr.s_addr = INADDR_ANY;
+ }
+ if (bind(listen_sd, (struct sockaddr *) &serveraddr, sizeof(serveraddr)) == -1) {
+ perror("bind() failed");
+ exit(1);
+ }
+ printf("Started flookup server on %s port %i\n", inet_ntoa(serveraddr.sin_addr), port_number); fflush(stdout);
+}
diff --git a/foma.c b/foma.c
new file mode 100644
index 0000000..ee0a0f9
--- /dev/null
+++ b/foma.c
@@ -0,0 +1,243 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <ctype.h>
+#include <unistd.h>
+#include <getopt.h>
+#include <time.h>
+#include <readline/readline.h>
+#include "foma.h"
+
+/* Front-end behavior variables */
+int pipe_mode = 0;
+int quiet_mode = 0;
+static int use_readline = 1;
+
+int promptmode = PROMPT_MAIN;
+int apply_direction;
+
+/* Variable to pass the position of rl completion to our completer */
+static int smatch;
+
+char *usagestring = "Usage: foma [-e \"command\"] [-f run-once-script] [-l startupscript] [-p] [-q] [-s] [-v]\n";
+
+static char** my_completion(const char*, int ,int);
+char *my_generator(const char* , int);
+char *cmd [] = {"ambiguous upper","apply down","apply med","apply up","apropos","assert-stack","clear stack","close sigma","compact sigma","complete net","compose net","concatenate net","crossproduct net","define","determinize net","echo","eliminate flags","eliminate flag","export cmatrix","extract ambiguous","extract unambiguous","factorize","help license","help warranty","ignore net","intersect net","invert net","label net","letter machine","load defined","lower-side net","minimize net [...]
+
+char *abbrvcmd [] = {"ambiguous","close","down","up","med","size","loadd","lower-words","upper-words","net","random-lower","random-upper","words","random-words","regex","rpl","au revoir","bye","exit","saved","seq","ss","stack","tunam","tid","tfu","tlu","tuu","tnu","tnn","tseq","tsf","equ","pss","psz","ratt","tfd","hyvästi","watt","wpl","examb","exunamb","pairs","random-pairs",NULL};
+
+/* #include "yy.tab.h" */
+
+int view_net(struct fsm *net);
+
+extern int input_is_file;
+extern int add_history (const char *);
+extern int my_yyparse(char *my_string);
+void print_help();
+void xprintf(char *string) { return ; printf("%s",string); }
+char disclaimer[] = "Foma, version 0.9.18alpha\nCopyright © 2008-2015 Mans Hulden\nThis is free software; see the source code for copying conditions.\nThere is ABSOLUTELY NO WARRANTY; for details, type \"help license\"\n\nType \"help\" to list all commands available.\nType \"help <topic>\" or help \"<operator>\" for further help.\n\n";
+
+/* A static variable for holding the line. */
+
+static char *command = (char *)NULL;
+char *flex_command = NULL;
+static char *line_read = (char *)NULL;
+char no_readline_line[512];
+
+/* Read a string, and return a pointer to it.
+ Returns NULL on EOF. */
+
+char *rl_gets(char *prompt) {
+
+ /* If the buffer has already been allocated,
+ return the memory to the free pool. */
+ if (use_readline == 1) {
+ if (line_read) {
+ free(line_read);
+ line_read = (char *)NULL;
+ }
+ }
+ if (use_readline == 0) {
+ printf("%s",prompt);
+ line_read = fgets(no_readline_line, 511, stdin);
+ if (line_read != NULL) {
+ strip_newline(line_read);
+ }
+ } else {
+ line_read = readline(prompt);
+ }
+
+ /* If the line has any text in it,
+ save it on the history. */
+ if (use_readline == 1) {
+ if (line_read && *line_read)
+ add_history(line_read);
+ }
+ return (line_read);
+}
+
+int main(int argc, char *argv[]) {
+ int opt;
+
+ char *scriptfile, prompt[50];
+ extern void my_interfaceparse(char *my_string);
+ /* YY_BUFFER_STATE flex_command; */
+ stack_init();
+ srand ((unsigned int)time(NULL));
+ /* Init defined_networks structures */
+ g_defines = defined_networks_init();
+ g_defines_f = defined_functions_init();
+
+ while ((opt = getopt(argc, argv, "e:f:hl:pqrsv")) != -1) {
+ switch(opt) {
+ case 'e':
+ my_interfaceparse(optarg);
+ break;
+ case 'f':
+ scriptfile = file_to_mem(optarg);
+ if (scriptfile != NULL) {
+ input_is_file = 1;
+ my_interfaceparse(scriptfile);
+ }
+ exit(0);
+ case 'h':
+ print_help();
+ exit(0);
+ case 'l':
+ scriptfile = file_to_mem(optarg);
+ if (scriptfile != NULL) {
+ input_is_file = 1;
+ my_interfaceparse(scriptfile);
+ xxfree(scriptfile);
+ }
+ break;
+ case 'p':
+ pipe_mode = 1;
+ break;
+ case 'q':
+ quiet_mode = 1;
+ break;
+ case 'r':
+ use_readline = 0;
+ break;
+ case 's':
+ exit(0);
+ case 'v':
+ printf("%s %i.%i.%i%s\n",argv[0],MAJOR_VERSION,MINOR_VERSION,BUILD_VERSION,STATUS_VERSION);
+ exit(0);
+ default:
+ fprintf(stderr, "%s", usagestring);
+ exit(EXIT_FAILURE);
+ }
+ }
+
+ if (!pipe_mode && !quiet_mode)
+ printf("%s",disclaimer);
+ rl_basic_word_break_characters = " >";
+
+ rl_attempted_completion_function = my_completion;
+ for(;;) {
+ if (promptmode == PROMPT_MAIN)
+ sprintf(prompt, "foma[%i]: ",stack_size());
+ if (promptmode == PROMPT_A && apply_direction == AP_D)
+ sprintf(prompt, "apply down> ");
+ if (promptmode == PROMPT_A && apply_direction == AP_U)
+ sprintf(prompt, "apply up> ");
+ if (promptmode == PROMPT_A && apply_direction == AP_M)
+ sprintf(prompt, "apply med> ");
+ if (pipe_mode || quiet_mode)
+ prompt[0] = '\0';
+
+ fflush(stdout);
+
+ command = rl_gets(prompt);
+
+ if (command == NULL && promptmode == PROMPT_MAIN) {
+ printf("\n");
+ exit(0);
+ }
+ if (command == NULL && promptmode == PROMPT_A) {
+ /* apply_clear(); */
+ promptmode = PROMPT_MAIN;
+ printf("\n");
+ continue;
+ }
+ input_is_file = 0;
+ my_interfaceparse(command);
+ }
+}
+
+void print_help() {
+ printf("%s",usagestring);
+ printf("Options:\n");
+ printf("-e \"command\"\texecute a command on startup (-e can be invoked several times)\n");
+ printf("-f scriptfile\tread commands from scriptfile on startup, and quit\n");
+ printf("-l scriptfile\tread commands from scriptfile on startup\n");
+ printf("-p\t\tpipe-mode\n");
+ printf("-q\t\tquiet mode (more quiet than pipe-mode)\n");
+ printf("-r\t\tdon't use readline library for input\n");
+ printf("-s\t\tstop execution and exit\n");
+ printf("-v\t\tprint version number\n");
+}
+
+static char **my_completion(const char *text, int start, int end) {
+ char **matches;
+
+ matches = (char **)NULL;
+ smatch = start;
+ matches = rl_completion_matches ((char*)text, &my_generator);
+
+ return (matches);
+}
+
+char *my_generator(const char *text, int state) {
+ static int list_index, list_index2, len, nummatches;
+ char *name;
+ text = rl_line_buffer;
+ if (!state) {
+ list_index = 0;
+ list_index2 = 0;
+ nummatches = 0;
+ len = strlen(text);
+ }
+
+ while ((name = cmd[list_index])) {
+ list_index++;
+
+ if (strncmp (name, text, len) == 0) {
+ nummatches++;
+ /* Can't use xxstrdup here */
+ return(strdup(name+smatch));
+ }
+ }
+
+ if (rl_point > 0) {
+ while ((name = abbrvcmd[list_index2])) {
+ list_index2++;
+
+ /* Can't use xxstrdup here */
+ if (strncmp (name, text, len) == 0)
+ return(strdup(name+smatch));
+ }
+ }
+
+ /* If no names matched, then return NULL. */
+ return ((char *)NULL);
+}
diff --git a/foma.h b/foma.h
new file mode 100644
index 0000000..362187b
--- /dev/null
+++ b/foma.h
@@ -0,0 +1,161 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include "fomalib.h"
+
+#define AP_D 1 /* Apply down */
+#define AP_U 2 /* Apply up */
+#define AP_M 3 /* Apply minimum edit distance */
+
+#define PROMPT_MAIN 0 /* Regular prompt */
+#define PROMPT_A 1 /* Apply prompt */
+
+struct defined_networks *g_defines;
+struct defined_functions *g_defines_f;
+
+/** User stack */
+struct stack_entry {
+ int number;
+ struct apply_handle *ah;
+ struct apply_med_handle *amedh;
+ struct fsm *fsm;
+ struct stack_entry *next;
+ struct stack_entry *previous;
+};
+
+/* Quantifier & Logic-related */
+char *find_quantifier(char *string);
+void add_quantifier (char *string);
+void purge_quantifier (char *string);
+struct fsm *union_quantifiers();
+int count_quantifiers();
+void clear_quantifiers();
+
+/* Main Stack functions */
+int stack_add(struct fsm *fsm);
+int stack_size();
+int stack_init();
+struct fsm *stack_pop();
+int stack_isempty();
+int stack_turn();
+struct stack_entry *stack_find_top();
+struct stack_entry *stack_find_second();
+struct stack_entry *stack_find_bottom();
+int stack_clear();
+int stack_rotate();
+int stack_print();
+struct apply_handle *stack_get_ah();
+struct apply_med_handle *stack_get_med_ah();
+
+/* Iface */
+void iface_ambiguous_upper(void);
+void iface_apply_down(char *word);
+int iface_apply_file(char *infilename, char *outfilename, int direction);
+void iface_apply_med(char *word);
+void iface_apply_set_params(struct apply_handle *h);
+void iface_apply_up(char *word);
+void iface_apropos(char *s);
+void iface_close(void);
+void iface_compact(void);
+void iface_complete(void);
+void iface_compose(void);
+void iface_conc(void);
+void iface_crossproduct(void);
+void iface_determinize(void);
+void iface_eliminate_flags(void);
+void iface_eliminate_flag(char *name);
+int iface_extract_number(char *s);
+void iface_extract_ambiguous(void);
+void iface_extract_unambiguous(void);
+void iface_factorize(void);
+void iface_help_search(char *s);
+void iface_help(void);
+void iface_ignore(void);
+void iface_intersect(void);
+void iface_invert(void);
+void iface_load_defined(char *filename);
+void iface_load_stack(char *filename);
+void iface_lower_side(void);
+void iface_minimize(void);
+void iface_one_plus(void);
+void iface_pop(void);
+void iface_label_net(void);
+void iface_letter_machine(void);
+void iface_lower_words(int limit);
+void iface_name_net(char *name);
+void iface_negate(void);
+void iface_print_cmatrix(void);
+void iface_print_cmatrix_att(char *filename);
+void iface_print_net(char *netname, char *filename);
+void iface_print_defined(void);
+void iface_print_dot(char *filename);
+void iface_print_shortest_string();
+void iface_print_shortest_string_size();
+void iface_print_name(void);
+void iface_quit(void);
+void iface_apply_random(char *(*applyer)(), int limit);
+void iface_random_lower(int limit);
+void iface_random_upper(int limit);
+void iface_random_words(int limit);
+void iface_pairs(int limit);
+void iface_pairs_file(char *filename);
+void iface_random_pairs(int limit);
+void iface_print_sigma(void);
+void iface_print_stats(void);
+void iface_shuffle(void);
+void iface_sort(void);
+void iface_sort_input(void);
+void iface_sort_output(void);
+int iface_stack_check(int size);
+void iface_upper_words(int limit);
+void iface_prune(void);
+int iface_read_att(char *filename);
+int iface_read_prolog(char *filename);
+int iface_read_spaced_text(char *filename);
+int iface_read_text(char *filename);
+void iface_reverse(void);
+void iface_rotate(void);
+void iface_save_defined(char *filename);
+void iface_save_stack(char *filename);
+void iface_sequentialize(void);
+void iface_set_variable(char *name, char *value);
+void iface_show_variables(void);
+void iface_show_variable(char *name);
+void iface_sigma_net();
+void iface_substitute_defined (char *original, char *substitute);
+void iface_substitute_symbol (char *original, char *substitute);
+void iface_test_equivalent(void);
+void iface_test_functional(void);
+void iface_test_identity(void);
+void iface_test_lower_universal(void);
+void iface_test_sequential(void);
+void iface_test_unambiguous(void);
+void iface_test_upper_universal(void);
+void iface_test_nonnull(void);
+void iface_test_null(void);
+void iface_turn(void);
+void iface_twosided_flags(void);
+void iface_union(void);
+void iface_upper_side(void);
+void iface_view(void);
+void iface_warranty(void);
+void iface_words(int limit);
+void iface_words_file(char *filename, int type);
+int iface_write_att(char *filename);
+void iface_write_prolog(char *filename);
+void iface_zero_plus(void);
+int print_stats(struct fsm *net);
diff --git a/fomalib.h b/fomalib.h
new file mode 100644
index 0000000..9d1102a
--- /dev/null
+++ b/fomalib.h
@@ -0,0 +1,505 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+#include <stdio.h>
+#include <inttypes.h>
+#include <string.h>
+#include "zlib.h"
+
+#define INLINE inline
+
+#define FEXPORT __attribute__((visibility("default")))
+
+/* Library version */
+#define MAJOR_VERSION 0
+#define MINOR_VERSION 9
+#define BUILD_VERSION 18
+#define STATUS_VERSION "alpha"
+
+/* Special symbols on arcs */
+#define EPSILON 0
+#define UNKNOWN 1
+#define IDENTITY 2
+
+/* Variants of ignore operation */
+#define OP_IGNORE_ALL 1
+#define OP_IGNORE_INTERNAL 2
+
+/* Replacement direction */
+#define OP_UPWARD_REPLACE 1
+#define OP_RIGHTWARD_REPLACE 2
+#define OP_LEFTWARD_REPLACE 3
+#define OP_DOWNWARD_REPLACE 4
+#define OP_TWO_LEVEL_REPLACE 5
+
+/* Arrow types in fsmrules */
+#define ARROW_RIGHT 1
+#define ARROW_LEFT 2
+#define ARROW_OPTIONAL 4
+#define ARROW_DOTTED 8 /* This is for the [..] part of a dotted rule */
+#define ARROW_LONGEST_MATCH 16
+#define ARROW_SHORTEST_MATCH 32
+#define ARROW_LEFT_TO_RIGHT 64
+#define ARROW_RIGHT_TO_LEFT 128
+
+/* Flag types */
+#define FLAG_UNIFY 1
+#define FLAG_CLEAR 2
+#define FLAG_DISALLOW 4
+#define FLAG_NEGATIVE 8
+#define FLAG_POSITIVE 16
+#define FLAG_REQUIRE 32
+#define FLAG_EQUAL 64
+
+#define NO 0
+#define YES 1
+#define UNK 2
+
+#define PATHCOUNT_CYCLIC -1
+#define PATHCOUNT_OVERFLOW -2
+#define PATHCOUNT_UNKNOWN -3
+
+#define M_UPPER 1
+#define M_LOWER 2
+
+#define APPLY_INDEX_INPUT 1
+#define APPLY_INDEX_OUTPUT 2
+
+/* Defined networks */
+struct defined_networks {
+ char *name;
+ struct fsm *net;
+ struct defined_networks *next;
+};
+
+/* Defined functions */
+struct defined_functions {
+ char *name;
+ char *regex;
+ int numargs;
+ struct defined_functions *next;
+};
+
+struct defined_quantifiers {
+ char *name;
+ struct defined_quantifiers *next;
+};
+
+/* Automaton structures */
+
+/** Main automaton structure */
+struct fsm {
+ char name[40];
+ int arity;
+ int arccount;
+ int statecount;
+ int linecount;
+ int finalcount;
+ long long pathcount;
+ int is_deterministic;
+ int is_pruned;
+ int is_minimized;
+ int is_epsilon_free;
+ int is_loop_free;
+ int is_completed;
+ int arcs_sorted_in;
+ int arcs_sorted_out;
+ struct fsm_state *states; /* pointer to first line */
+ struct sigma *sigma;
+ struct medlookup *medlookup;
+};
+
+/* Minimum edit distance structure */
+
+struct medlookup {
+ int *confusion_matrix; /* Confusion matrix */
+};
+
+/** Array of states */
+struct fsm_state {
+ int state_no; /* State number */
+ short int in ;
+ short int out ;
+ int target;
+ char final_state ;
+ char start_state ;
+};
+
+struct fsmcontexts {
+ struct fsm *left;
+ struct fsm *right;
+ struct fsmcontexts *next;
+ struct fsm *cpleft; /* Only used internally when compiling rewrite rules */
+ struct fsm *cpright; /* ditto */
+};
+
+struct fsmrules {
+ struct fsm *left;
+ struct fsm *right;
+ struct fsm *right2; /*Only needed for A -> B ... C rules*/
+ struct fsm *cross_product;
+ struct fsmrules *next;
+ int arrow_type;
+ int dotted; /* [.A.] rule */
+};
+
+struct rewrite_set {
+ struct fsmrules *rewrite_rules;
+ struct fsmcontexts *rewrite_contexts;
+ struct rewrite_set *next;
+ int rule_direction; /* || \\ // \/ */
+};
+
+FEXPORT void fsm_clear_contexts(struct fsmcontexts *contexts);
+
+/** Linked list of sigma */
+/** number < IDENTITY is reserved for special symbols */
+struct sigma {
+ int number;
+ char *symbol;
+ struct sigma *next;
+};
+
+#include "fomalibconf.h"
+
+/* Define functions */
+FEXPORT struct defined_networks *defined_networks_init(void);
+FEXPORT struct defined_functions *defined_functions_init(void);
+struct fsm *find_defined(struct defined_networks *def, char *string);
+char *find_defined_function(struct defined_functions *deff, char *name, int numargs);
+FEXPORT int add_defined(struct defined_networks *def, struct fsm *net, char *string);
+FEXPORT int add_defined_function (struct defined_functions *deff, char *name, char *regex, int numargs);
+int remove_defined (struct defined_networks *def, char *string);
+
+/********************/
+/* Basic operations */
+/********************/
+
+FEXPORT char *fsm_get_library_version_string();
+
+FEXPORT struct fsm *fsm_determinize(struct fsm *net);
+FEXPORT struct fsm *fsm_epsilon_remove(struct fsm *net);
+FEXPORT struct fsm *fsm_find_ambiguous(struct fsm *net, int **extras);
+FEXPORT struct fsm *fsm_minimize(struct fsm *net);
+FEXPORT struct fsm *fsm_coaccessible(struct fsm *net);
+FEXPORT struct fsm *fsm_topsort(struct fsm *net);
+FEXPORT void fsm_sort_arcs(struct fsm *net, int direction);
+FEXPORT struct fsm *fsm_mark_ambiguous(struct fsm *net);
+FEXPORT struct fsm *fsm_sequentialize(struct fsm *net);
+FEXPORT struct fsm *fsm_bimachine(struct fsm *net);
+
+FEXPORT struct fsm *fsm_parse_regex(char *regex, struct defined_networks *defined_nets, struct defined_functions *defined_funcs);
+FEXPORT struct fsm *fsm_reverse(struct fsm *net);
+FEXPORT struct fsm *fsm_invert(struct fsm *net);
+FEXPORT struct fsm *fsm_lower(struct fsm *net);
+FEXPORT struct fsm *fsm_upper(struct fsm *net);
+FEXPORT struct fsm *fsm_kleene_star(struct fsm *net);
+FEXPORT struct fsm *fsm_kleene_plus(struct fsm *net);
+FEXPORT struct fsm *fsm_optionality(struct fsm *net);
+FEXPORT struct fsm *fsm_boolean(int value);
+FEXPORT struct fsm *fsm_concat(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_concat_n(struct fsm *net1, int n);
+FEXPORT struct fsm *fsm_concat_m_n(struct fsm *net1, int m, int n);
+FEXPORT struct fsm *fsm_union(struct fsm *net_1, struct fsm *net_2);
+FEXPORT struct fsm *fsm_priority_union_upper(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_priority_union_lower(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_intersect(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_compose(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_lenient_compose(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_cross_product(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_shuffle(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_precedes(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_follows(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_symbol(char *symbol);
+FEXPORT struct fsm *fsm_explode(char *symbol);
+FEXPORT struct fsm *fsm_escape(char *symbol);
+FEXPORT struct fsm *fsm_copy(struct fsm *net);
+FEXPORT struct fsm *fsm_complete(struct fsm *net);
+FEXPORT struct fsm *fsm_complement(struct fsm *net);
+FEXPORT struct fsm *fsm_term_negation(struct fsm *net1);
+FEXPORT struct fsm *fsm_minus(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_simple_replace(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_context_restrict(struct fsm *X, struct fsmcontexts *LR);
+FEXPORT struct fsm *fsm_contains(struct fsm *net);
+FEXPORT struct fsm *fsm_contains_opt_one(struct fsm *net);
+FEXPORT struct fsm *fsm_contains_one(struct fsm *net);
+FEXPORT struct fsm *fsm_ignore(struct fsm *net1, struct fsm *net2, int operation);
+FEXPORT struct fsm *fsm_quotient_right(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_quotient_left(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_quotient_interleave(struct fsm *net1, struct fsm *net2);
+FEXPORT struct fsm *fsm_substitute_label(struct fsm *net, char *original, struct fsm *substitute);
+FEXPORT struct fsm *fsm_substitute_symbol(struct fsm *net, char *original, char *substitute);
+FEXPORT struct fsm *fsm_universal();
+FEXPORT struct fsm *fsm_empty_set();
+FEXPORT struct fsm *fsm_empty_string();
+FEXPORT struct fsm *fsm_identity();
+FEXPORT struct fsm *fsm_quantifier(char *string);
+FEXPORT struct fsm *fsm_logical_eq(char *string1, char *string2);
+FEXPORT struct fsm *fsm_logical_precedence(char *string1, char *string2);
+FEXPORT struct fsm *fsm_lowerdet(struct fsm *net);
+FEXPORT struct fsm *fsm_lowerdeteps(struct fsm *net);
+FEXPORT struct fsm *fsm_markallfinal(struct fsm *net);
+FEXPORT struct fsm *fsm_extract_nonidentity(struct fsm *net);
+FEXPORT struct fsm *fsm_extract_ambiguous_domain(struct fsm *net);
+FEXPORT struct fsm *fsm_extract_ambiguous(struct fsm *net);
+FEXPORT struct fsm *fsm_extract_unambiguous(struct fsm *net);
+FEXPORT struct fsm *fsm_sigma_net(struct fsm *net);
+FEXPORT struct fsm *fsm_sigma_pairs_net(struct fsm *net);
+FEXPORT struct fsm *fsm_equal_substrings(struct fsm *net, struct fsm *left, struct fsm *right);
+FEXPORT struct fsm *fsm_letter_machine(struct fsm *net);
+FEXPORT struct fsm *fsm_mark_fsm_tail(struct fsm *net, struct fsm *marker);
+FEXPORT struct fsm *fsm_add_loop(struct fsm *net, struct fsm *marker, int finals);
+FEXPORT struct fsm *fsm_add_sink(struct fsm *net, int final);
+FEXPORT struct fsm *fsm_left_rewr(struct fsm *net, struct fsm *rewr);
+FEXPORT struct fsm *fsm_flatten(struct fsm *net, struct fsm *epsilon);
+FEXPORT struct fsm *fsm_unflatten(struct fsm *net, char *epsilon_sym, char *repeat_sym);
+FEXPORT struct fsm *fsm_close_sigma(struct fsm *net, int mode);
+FEXPORT char *fsm_network_to_char(struct fsm *net);
+
+/* Remove those symbols from sigma that have the same distribution as IDENTITY */
+FEXPORT void fsm_compact(struct fsm *net);
+
+FEXPORT int flag_build(int ftype, char *fname, char *fvalue, int fftype, char *ffname, char *ffvalue);
+
+/* Eliminate flag diacritics and return equivalent FSM */
+/* with name = NULL the function eliminates all flag diacritics */
+FEXPORT struct fsm *flag_eliminate(struct fsm *net, char *name);
+
+/* Enforce twosided flag diacritics */
+FEXPORT struct fsm *flag_twosided(struct fsm *net);
+
+/* Compile a rewrite rule */
+FEXPORT struct fsm *fsm_rewrite();
+
+/* Boolean tests */
+FEXPORT int fsm_isempty(struct fsm *net);
+FEXPORT int fsm_isfunctional(struct fsm *net);
+FEXPORT int fsm_isunambiguous(struct fsm *net);
+FEXPORT int fsm_isidentity(struct fsm *net);
+FEXPORT int fsm_isuniversal(struct fsm *net);
+FEXPORT int fsm_issequential(struct fsm *net);
+FEXPORT int fsm_equivalent(struct fsm *net1, struct fsm *net2);
+
+/* Test if a symbol occurs in a FSM */
+/* side = M_UPPER (upper side) M_LOWER (lower side), M_UPPER+M_LOWER (both) */
+FEXPORT int fsm_symbol_occurs(struct fsm *net, char *symbol, int side);
+
+/* Merges two alphabets destructively */
+FEXPORT void fsm_merge_sigma(struct fsm *net1, struct fsm *net2);
+
+/* Copies an alphabet */
+
+FEXPORT struct sigma *sigma_copy(struct sigma *sigma);
+
+/* Create empty FSM */
+FEXPORT struct fsm *fsm_create(char *name);
+FEXPORT struct fsm_state *fsm_empty();
+
+/* Frees alphabet */
+FEXPORT int fsm_sigma_destroy(struct sigma *sigma);
+
+/* Frees a FSM, associated data such as alphabet and confusion matrix */
+FEXPORT int fsm_destroy(struct fsm *net);
+
+/* IO functions */
+FEXPORT struct fsm *read_att(char *filename);
+FEXPORT int net_print_att(struct fsm *net, FILE *outfile);
+FEXPORT struct fsm *fsm_read_prolog(char *filename);
+FEXPORT char *file_to_mem(char *name);
+FEXPORT struct fsm *fsm_read_binary_file(char *filename);
+FEXPORT struct fsm *fsm_read_binary_file_multiple(fsm_read_binary_handle fsrh);
+FEXPORT fsm_read_binary_handle fsm_read_binary_file_multiple_init(char *filename);
+FEXPORT struct fsm *fsm_read_text_file(char *filename);
+FEXPORT struct fsm *fsm_read_spaced_text_file(char *filename);
+FEXPORT int fsm_write_binary_file(struct fsm *net, char *filename);
+FEXPORT int load_defined(struct defined_networks *def, char *filename);
+FEXPORT int save_defined(struct defined_networks *def, char *filename);
+FEXPORT int save_stack_att();
+FEXPORT int foma_write_prolog(struct fsm *net, char *filename);
+FEXPORT int foma_net_print(struct fsm *net, gzFile outfile);
+
+/* Lookups */
+
+/* Frees memory alloced by apply_init */
+FEXPORT void apply_clear(struct apply_handle *h);
+/* To be called before applying words */
+FEXPORT struct apply_handle *apply_init(struct fsm *net);
+FEXPORT struct apply_med_handle *apply_med_init(struct fsm *net);
+FEXPORT void apply_med_clear(struct apply_med_handle *h);
+
+FEXPORT void apply_med_set_heap_max(struct apply_med_handle *medh, int max);
+FEXPORT void apply_med_set_med_limit(struct apply_med_handle *medh, int max);
+FEXPORT void apply_med_set_med_cutoff(struct apply_med_handle *medh, int max);
+FEXPORT int apply_med_get_cost(struct apply_med_handle *medh);
+FEXPORT void apply_med_set_align_symbol(struct apply_med_handle *medh, char *align);
+FEXPORT char *apply_med_get_instring(struct apply_med_handle *medh);
+FEXPORT char *apply_med_get_outstring(struct apply_med_handle *medh);
+
+FEXPORT char *apply_down(struct apply_handle *h, char *word);
+FEXPORT char *apply_up(struct apply_handle *h, char *word);
+FEXPORT char *apply_med(struct apply_med_handle *medh, char *word);
+FEXPORT char *apply_upper_words(struct apply_handle *h);
+FEXPORT char *apply_lower_words(struct apply_handle *h);
+FEXPORT char *apply_words(struct apply_handle *h);
+FEXPORT char *apply_random_lower(struct apply_handle *h);
+FEXPORT char *apply_random_upper(struct apply_handle *h);
+FEXPORT char *apply_random_words(struct apply_handle *h);
+/* Reset the iterator to start anew with enumerating functions */
+FEXPORT void apply_reset_enumerator(struct apply_handle *h);
+FEXPORT void apply_index(struct apply_handle *h, int inout, int densitycutoff, int mem_limit, int flags_only);
+FEXPORT void apply_set_show_flags(struct apply_handle *h, int value);
+FEXPORT void apply_set_obey_flags(struct apply_handle *h, int value);
+FEXPORT void apply_set_print_space(struct apply_handle *h, int value);
+FEXPORT void apply_set_print_pairs(struct apply_handle *h, int value);
+FEXPORT void apply_set_space_symbol(struct apply_handle *h, char *space);
+FEXPORT void apply_set_separator(struct apply_handle *h, char *symbol);
+FEXPORT void apply_set_epsilon(struct apply_handle *h, char *symbol);
+
+/* Minimum edit distance & spelling correction */
+FEXPORT void fsm_create_letter_lookup(struct apply_med_handle *medh, struct fsm *net);
+FEXPORT void cmatrix_init(struct fsm *net);
+FEXPORT void cmatrix_default_substitute(struct fsm *net, int cost);
+FEXPORT void cmatrix_default_insert(struct fsm *net, int cost);
+FEXPORT void cmatrix_default_delete(struct fsm *net, int cost);
+FEXPORT void cmatrix_set_cost(struct fsm *net, char *in, char *out, int cost);
+FEXPORT void cmatrix_print(struct fsm *net);
+FEXPORT void cmatrix_print_att(struct fsm *net, FILE *outfile);
+
+/* Lexc */
+ FEXPORT struct fsm *fsm_lexc_parse_file(char *myfile, int verbose);
+ FEXPORT struct fsm *fsm_lexc_parse_string(char *mystring, int verbose);
+
+/*************************/
+/* Construction routines */
+/*************************/
+
+FEXPORT struct fsm_construct_handle *fsm_construct_init(char *name);
+FEXPORT void fsm_construct_set_final(struct fsm_construct_handle *handle, int state_no);
+FEXPORT void fsm_construct_set_initial(struct fsm_construct_handle *handle, int state_no);
+FEXPORT void fsm_construct_add_arc(struct fsm_construct_handle *handle, int source, int target, char *in, char *out);
+FEXPORT void fsm_construct_add_arc_nums(struct fsm_construct_handle *handle, int source, int target, int in, int out);
+FEXPORT int fsm_construct_add_symbol(struct fsm_construct_handle *handle, char *symbol);
+FEXPORT int fsm_construct_check_symbol(struct fsm_construct_handle *handle, char *symbol);
+FEXPORT void fsm_construct_copy_sigma(struct fsm_construct_handle *handle, struct sigma *sigma);
+FEXPORT struct fsm *fsm_construct_done(struct fsm_construct_handle *handle);
+
+
+/******************/
+/* String hashing */
+/******************/
+
+struct sh_handle {
+ struct sh_hashtable *hash;
+ int lastvalue;
+};
+
+struct sh_hashtable {
+ char *string;
+ int value;
+ struct sh_hashtable *next;
+};
+
+struct sh_handle *sh_init();
+void sh_done(struct sh_handle *sh);
+char *sh_find_string(struct sh_handle *sh, char *string);
+char *sh_find_add_string(struct sh_handle *sh, char *string, int value);
+char *sh_add_string(struct sh_handle *sh, char *string, int value);
+int sh_get_value(struct sh_handle *sh);
+
+/*********************/
+/* Trie construction */
+/*********************/
+
+struct trie_hash {
+ char *insym;
+ char *outsym;
+ unsigned int sourcestate;
+ unsigned int targetstate;
+ struct trie_hash *next;
+};
+
+struct trie_states {
+ _Bool is_final;
+};
+
+struct fsm_trie_handle {
+ struct trie_states *trie_states;
+ unsigned int trie_cursor;
+ struct trie_hash *trie_hash;
+ unsigned int used_states;
+ unsigned int statesize;
+ struct sh_handle *sh_hash;
+};
+
+FEXPORT struct fsm_trie_handle *fsm_trie_init();
+FEXPORT struct fsm *fsm_trie_done(struct fsm_trie_handle *th);
+FEXPORT void fsm_trie_add_word(struct fsm_trie_handle *th, char *word);
+FEXPORT void fsm_trie_end_word(struct fsm_trie_handle *th);
+FEXPORT void fsm_trie_symbol(struct fsm_trie_handle *th, char *insym, char *outsym);
+
+/***********************/
+/* Extraction routines */
+/***********************/
+
+struct fsm_read_handle {
+ struct fsm_state *arcs_head;
+ struct fsm_state **states_head;
+ struct fsm_state *arcs_cursor;
+ int *finals_head;
+ int *finals_cursor;
+ struct fsm_state **states_cursor;
+ int *initials_head;
+ int *initials_cursor;
+ int current_state;
+ struct fsm_sigma_list *fsm_sigma_list;
+ int sigma_list_size;
+ struct fsm *net;
+ unsigned char *lookuptable;
+ _Bool has_unknowns;
+};
+
+FEXPORT struct fsm_read_handle *fsm_read_init(struct fsm *net);
+FEXPORT void fsm_read_reset(struct fsm_read_handle *handle);
+FEXPORT int fsm_read_is_final(struct fsm_read_handle *h, int state);
+FEXPORT int fsm_read_is_initial(struct fsm_read_handle *h, int state);
+FEXPORT int fsm_get_num_states(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_has_unknowns(struct fsm_read_handle *handle);
+/* Move iterator one arc forward. Returns 0 on no more arcs */
+FEXPORT int fsm_get_next_arc(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_arc_source(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_arc_target(struct fsm_read_handle *handle);
+FEXPORT char *fsm_get_arc_in(struct fsm_read_handle *handle);
+FEXPORT char *fsm_get_arc_out(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_arc_num_in(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_arc_num_out(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_symbol_number(struct fsm_read_handle *handle, char *symbol);
+
+/* Iterates over initial and final states, returns -1 on end */
+FEXPORT int fsm_get_next_initial(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_next_final(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_next_state(struct fsm_read_handle *handle);
+FEXPORT int fsm_get_next_state_arc(struct fsm_read_handle *handle);
+
+/* Frees memory associated with a read handle */
+FEXPORT void fsm_read_done(struct fsm_read_handle *handle);
+
+#ifdef __cplusplus
+}
+#endif
diff --git a/fomalibconf.h b/fomalibconf.h
new file mode 100644
index 0000000..b000b82
--- /dev/null
+++ b/fomalibconf.h
@@ -0,0 +1,305 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+struct state_array {
+ struct fsm_state *transitions;
+};
+
+struct fsm_trans_list {
+ short int in;
+ short int out;
+ int target;
+ struct fsm_trans_list *next;
+};
+
+struct fsm_state_list {
+ _Bool used;
+ _Bool is_final;
+ _Bool is_initial;
+ short int num_trans;
+ int state_number;
+ struct fsm_trans_list *fsm_trans_list;
+};
+
+struct fsm_sigma_list {
+ char *symbol;
+};
+
+struct fsm_sigma_hash {
+ char *symbol;
+ short int sym;
+ struct fsm_sigma_hash *next;
+};
+
+typedef void *fsm_read_binary_handle;
+
+struct fsm_construct_handle {
+ struct fsm_state_list *fsm_state_list;
+ int fsm_state_list_size;
+ struct fsm_sigma_list *fsm_sigma_list;
+ int fsm_sigma_list_size;
+ struct fsm_sigma_hash *fsm_sigma_hash;
+ int fsm_sigma_hash_size;
+ int maxstate;
+ int maxsigma;
+ int numfinals;
+ int hasinitial;
+ char *name;
+};
+
+struct apply_med_handle {
+ struct astarnode {
+ short int wordpos;
+ int fsmstate;
+ short int f;
+ short int g;
+ short int h;
+ int in;
+ int out;
+ int parent;
+ } *agenda;
+ int bytes_per_letter_array;
+ uint8_t *letterbits;
+ uint8_t *nletterbits;
+ int astarcount;
+ int heapcount;
+ int heap_size;
+ int agenda_size;
+ int maxdepth;
+ int maxsigma;
+ int wordlen;
+ int utf8len;
+ int cost;
+ int nummatches;
+ int curr_state;
+ int curr_g;
+ int curr_pos;
+ int lines;
+ int curr_agenda_offset;
+ int curr_node_has_match;
+ int med_limit;
+ int med_cutoff;
+ int med_max_heap_size;
+ int nodes_expanded;
+ int *cm;
+ char *word;
+ char *instring;
+ int instring_length;
+ char *outstring;
+ int outstring_length;
+ char *align_symbol;
+ int *heap;
+ int *intword;
+ struct sh_handle *sigmahash;
+ struct state_array *state_array;
+ struct fsm *net;
+ struct fsm_state *curr_ptr;
+ _Bool hascm;
+};
+
+struct apply_handle {
+
+ int ptr;
+ int curr_ptr;
+ int ipos;
+ int opos;
+ int mode;
+ int printcount;
+ int *numlines;
+ int *statemap;
+ int *marks;
+
+ struct sigma_trie {
+ int signum;
+ struct sigma_trie *next;
+ } *sigma_trie;
+
+ struct sigmatch_array {
+ int signumber ;
+ int consumes ;
+ } *sigmatch_array;
+
+ struct sigma_trie_arrays {
+ struct sigma_trie *arr;
+ struct sigma_trie_arrays *next;
+ } *sigma_trie_arrays;
+
+ int binsearch;
+ int indexed;
+ int state_has_index;
+ int sigma_size;
+ int sigmatch_array_size;
+ int current_instring_length;
+ int has_flags;
+ int obey_flags;
+ int show_flags;
+ int print_space;
+ char *space_symbol;
+ char *separator;
+ char *epsilon_symbol;
+ int print_pairs;
+ int apply_stack_ptr;
+ int apply_stack_top;
+ int oldflagneg;
+ int outstringtop;
+ int iterate_old;
+ int iterator;
+ uint8_t *flagstates;
+ char *outstring;
+ char *instring;
+ struct sigs {
+ char *symbol;
+ int length;
+ } *sigs;
+ char *oldflagvalue;
+
+ struct fsm *last_net;
+ struct fsm_state *gstates;
+ struct sigma *gsigma;
+ struct apply_state_index {
+ int fsmptr;
+ struct apply_state_index *next;
+ } **index_in, **index_out, *iptr;
+
+ struct flag_list {
+ char *name;
+ char *value;
+ short neg;
+ struct flag_list *next;
+ } *flag_list;
+
+ struct flag_lookup {
+ int type;
+ char *name;
+ char *value;
+ } *flag_lookup ;
+
+ struct searchstack {
+ int offset;
+ struct apply_state_index *iptr;
+ int state_has_index;
+ int opos;
+ int ipos;
+ int visitmark;
+ char *flagname;
+ char *flagvalue;
+ int flagneg;
+ } *searchstack ;
+};
+
+
+/* Automaton functions operating on fsm_state */
+int add_fsm_arc(struct fsm_state *fsm, int offset, int state_no, int in, int out, int target, int final_state, int start_state);
+struct fsm_state *fsm_state_copy(struct fsm_state *fsm_state, int linecount);
+
+/* Functions for constructing a FSM arc-by-arc */
+/* At the end of the constructions, the flags are updated automatically */
+
+/* Call fsm_state_init with the alphabet size to initialize the new machine */
+struct fsm_state *fsm_state_init(int sigma_size);
+
+/* Call set current state before calling fsm_state_add_arc */
+void fsm_state_set_current_state(int state_no, int final_state, int start_state);
+
+/* Add an arc */
+void fsm_state_add_arc(int state_no, int in, int out, int target, int final_state, int start_state);
+
+/* Call fsm_state_close() when done with arcs to a state */
+void fsm_state_close();
+
+/* Call this when done with entire FSM */
+void fsm_state_end_state();
+
+struct state_array *map_firstlines(struct fsm *net);
+
+FEXPORT void fsm_count(struct fsm *net);
+
+void fsm_sort_lines(struct fsm *net);
+void fsm_update_flags(struct fsm *net, int det, int pru, int min, int eps, int loop, int completed);
+
+int sort_cmp(const void *a, const void *b);
+
+int find_arccount(struct fsm_state *fsm);
+
+/* Internal int stack */
+int int_stack_isempty();
+int int_stack_isfull();
+void int_stack_clear();
+int int_stack_find (int entry);
+void int_stack_push(int c);
+int int_stack_pop();
+int int_stack_status();
+int int_stack_size();
+
+/* Internal ptr stack */
+int ptr_stack_isempty();
+void ptr_stack_clear();
+void *ptr_stack_pop();
+int ptr_stack_isfull();
+void ptr_stack_push(void *ptr);
+
+/* Sigma functions */
+FEXPORT int sigma_add (char *symbol, struct sigma *sigma);
+FEXPORT int sigma_add_number(struct sigma *sigma, char *symbol, int number);
+FEXPORT int sigma_add_special (int symbol, struct sigma *sigma);
+FEXPORT struct sigma *sigma_remove(char *symbol, struct sigma *sigma);
+FEXPORT struct sigma *sigma_remove_num(int num, struct sigma *sigma);
+
+int sigma_find (char *symbol, struct sigma *sigma);
+int sigma_find_number (int number, struct sigma *sigma);
+int sigma_substitute(char *orig, char *sub, struct sigma *sigma);
+FEXPORT char *sigma_string(int number, struct sigma *sigma);
+int sigma_sort (struct fsm *net);
+void sigma_cleanup (struct fsm *net, int force);
+FEXPORT struct sigma *sigma_create ();
+int sigma_size(struct sigma *sigma);
+FEXPORT int sigma_max(struct sigma *sigma);
+struct fsm_sigma_list *sigma_to_list(struct sigma *sigma);
+
+/* Debug */
+void xprintf(char *string);
+
+/* UTF8 */
+unsigned char *utf8code16tostr(char *str);
+int utf8iscombining(unsigned char *s);
+int utf8skip(char *str);
+int utf8strlen(char *str);
+int ishexstr(char *str);
+void decode_quoted(char *s);
+void dequote_string(char *s);
+char *remove_trailing(char *s, char c);
+char *escape_string(char *string, char chr);
+char *xstrrev(char *str);
+
+/* Flag-related */
+int flag_check(char *sm);
+char *flag_get_name(char *string);
+char *flag_get_value(char *string);
+int flag_get_type(char *string);
+
+/* Misc */
+char *trim(char *string);
+void strip_newline(char *s);
+char *streqrep(char *s, char *oldstring, char *newstring);
+char *xxstrndup(const char *s, size_t n);
+char *xxstrdup(const char *s);
+FEXPORT void *xxmalloc(size_t size);
+void *xxcalloc(size_t nmemb, size_t size);
+void *xxrealloc(void *ptr, size_t size);
+void xxfree(void *ptr);
+int next_power_of_two(int v);
+unsigned int round_up_to_power_of_two(unsigned int v);
diff --git a/iface.c b/iface.c
new file mode 100644
index 0000000..7a808c0
--- /dev/null
+++ b/iface.c
@@ -0,0 +1,1713 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <wchar.h>
+#include <stdlib.h>
+#include <string.h>
+#include <limits.h>
+#include <errno.h>
+#include <ctype.h>
+
+#include "foma.h"
+#include "zlib.h"
+
+extern int g_show_flags;
+extern int g_obey_flags;
+extern int g_flag_is_epsilon;
+extern int g_print_space;
+extern int g_print_pairs;
+extern int g_minimal;
+extern int g_name_nets;
+extern int g_print_sigma;
+extern int g_quit_on_fail;
+extern int g_quote_special;
+extern int g_recursive_define;
+extern int g_sort_arcs;
+extern int g_verbose;
+extern int g_minimize_hopcroft;
+extern int g_list_limit;
+extern int g_list_random_limit;
+extern int g_compose_tristate;
+extern int g_med_limit ;
+extern int g_med_cutoff ;
+extern int g_lexc_align ;
+extern char *g_att_epsilon;
+
+extern struct defined_networks *g_defines;
+extern struct defined_functions *g_defines_f;
+
+extern int foma_net_print(struct fsm *net, gzFile outfile);
+
+static char *sigptr(struct sigma *sigma, int number);
+static int print_dot(struct fsm *net, char *filename);
+static int print_net(struct fsm *net, char *filename);
+static int print_sigma(struct sigma *sigma, FILE *out);
+static int view_net(struct fsm *net);
+
+#define FVAR_BOOL 1
+#define FVAR_INT 2
+#define FVAR_STRING 3
+
+#define LINE_LIMIT 8192
+
+struct g_v {
+ void *ptr;
+ char *name;
+ int type;
+} global_vars[] = {
+ {&g_flag_is_epsilon, "flag-is-epsilon", FVAR_BOOL},
+ {&g_minimal, "minimal", FVAR_BOOL},
+ {&g_name_nets, "name-nets", FVAR_BOOL},
+ {&g_obey_flags, "obey-flags", FVAR_BOOL},
+ {&g_print_pairs, "print-pairs", FVAR_BOOL},
+ {&g_print_sigma, "print-sigma", FVAR_BOOL},
+ {&g_print_space, "print-space", FVAR_BOOL},
+ {&g_quit_on_fail, "quit-on-fail", FVAR_BOOL},
+ {&g_recursive_define, "recursive-define", FVAR_BOOL},
+ {&g_quote_special, "quote-special", FVAR_BOOL},
+ {&g_show_flags, "show-flags", FVAR_BOOL},
+ {&g_sort_arcs, "sort-arcs", FVAR_BOOL},
+ {&g_verbose, "verbose", FVAR_BOOL},
+ {&g_minimize_hopcroft,"hopcroft-min", FVAR_BOOL},
+ {&g_compose_tristate, "compose-tristate", FVAR_BOOL},
+ {&g_med_limit, "med-limit", FVAR_INT},
+ {&g_med_cutoff, "med-cutoff", FVAR_INT},
+ {&g_lexc_align, "lexc-align", FVAR_BOOL},
+ {&g_att_epsilon, "att-epsilon", FVAR_STRING},
+ {NULL, NULL, 0}
+};
+
+char warranty[] = "\nLicensed under the Apache License, Version 2.0 (the \"License\")\nyou may not use this file except in compliance with the License.\nYou may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n";
+
+struct global_help {
+ char *name;
+ char *help;
+ char *longhelp;
+} global_help[] = {
+ {"regex <regex>", "read a regular expression","Enter a regular expression and add result to top of stack.\nShort form: re\nSee `help operator' for operators, or `help precedence' for operator precedence."},
+ {"ambiguous upper","returns the input words which have multiple paths in a transducer","Short form: ambiguous\n"},
+ {"apply up <string>","apply <string> up to the top network on stack","Short form: up <string>\n"},
+ {"apply down <string>","apply <string> down to the top network on stack","Short form: down <string>\n" },
+ {"apply med <string>","find approximate matches to string in top network by minimum edit distance","Short form: med <string>\n" },
+ {"apply up","enter apply up mode (Ctrl-D exits)","Short form: up\n"},
+ {"apply down","enter apply down mode (Ctrl-D exits)","Short form: down\n"},
+ {"apply med","enter apply med mode (Ctrl-D exits)","Short form: med\n"},
+ {"apropos <string>","search help for <string>",""},
+ {"clear stack","clears the stack",""},
+ {"close sigma","removes unknown symbols from FSM","" },
+ {"compact sigma","removes redundant symbols from FSM","" },
+ {"complete net","completes the FSM","" },
+ {"compose net","composes networks on stack",""},
+ {"concatenate","concatenates networks on stack","" },
+ {"crossproduct net","cross-product of top two FSMs on stack","See ×\n" },
+ {"define <name> <r.e.>","define a network","Example: \ndefine A x -> y;\n and\nA = x -> y;\n\nare equivalent\n"},
+ {"define <fname>(<v1,..,vn>) <r.e.>","define function","Example: define Remove(X) [X -> 0].l;"},
+ {"determinize net","determinizes top FSM on stack",""},
+ {"echo <string>","echo a string",""},
+ {"eliminate flag <name>","eliminate flag <name> diacritics from the top network",""},
+ {"eliminate flags","eliminate all flag diacritics from the top network",""},
+ {"export cmatrix (filename)","export the confusion matrix as an AT&T transducer",""},
+ {"extract ambiguous","extracts the ambiguous paths of a transducer","Short form: examb"},
+ {"extract unambiguous","extracts the unambiguous paths of a transducer","Short form: exunamb"},
+ {"help license","prints license",""},
+ {"help warranty","prints warranty information",""},
+ {"ignore net","applies ignore to top two FSMs on stack","See /\n"},
+ {"intersect net","intersects FSMs on stack","See ∩ (or &)\n" },
+ {"invert net","inverts top FSM","See ⁻¹ (or .i)\n"},
+ {"label net","extracts all attested symbol pairs from FSM","See also: sigma net"},
+ {"letter machine","Converts top FSM to a letter machine","See also: _lm(L)"},
+ {"load stack <filename>","Loads networks and pushes them on the stack","Short form: load"},
+ {"load defined <filename>","Restores defined networks from file","Short form: loadd"},
+ {"lower-side net","takes lower projection of top FSM","See ₂ (or .l)\n"},
+ {"minimize net","minimizes top FSM","Minimization can be controlled through the variable minimal: when set to OFF FSMs are never minimized.\nAlso, hopcroft-min can be set to OFF in which case minimization is done by double reversal and determinization (aka Brzozowski's algorithm). It is likely to be much slower.\n"},
+ {"name net <string>","names top FSM",""},
+ {"negate net","complements top FSM","See ¬\n" },
+ {"one-plus net","Kleene plus on top FSM","See +\n" },
+ {"pop stack","remove top FSM from stack","" },
+ {"print cmatrix","prints the confusion matrix associated with the top network in tabular format",""},
+ {"print defined","prints defined symbols and functions",""},
+ {"print dot (>filename)","prints top FSM in Graphviz dot format",""},
+ {"print lower-words","prints words on the lower side of top FSM",""},
+ {"print lower-words > filename","prints words on the lower side of top FSM to file",""},
+ {"print name","prints the name of the top FSM","" },
+ {"print net","prints all information about top FSM","Short form: net\n" },
+ {"print pairs","prints input-output pairs from top FSM","Short form: pairs\n"},
+ {"print pairs > filename","prints input-output pairs from top FSM to file","Short form: pairs\n"},
+ {"print random-lower","prints random words from lower side","Short form: random-lower\n" },
+ {"print random-upper","prints random words from upper side","Short form: random-upper" },
+ {"print random-words","prints random words from top FSM","Short form: random-words\n"},
+ {"print random-pairs","prints random input-output pairs from top FSM","Short form: random-pairs\n"},
+ {"print sigma","prints the alphabet of the top FSM","Short form: sigma\n"},
+ {"print size","prints size information about top FSM","Short form: size\n"},
+ {"print shortest-string","prints the shortest string of the top FSM","Short form: pss\n"},
+ {"print shortest-string-size","prints length of shortest string","Short form: psz\n"},
+ {"print upper-words","prints words on the upper side of top FSM","Short form: upper-words"},
+ {"print upper-words > filename","prints words on the upper side of top FSM to file","Short form:upper-words"},
+ {"print words","prints words of top FSM","Short form: words"},
+ {"print words > filename","prints words of top FSM to file","Short form: words"},
+ {"prune net","makes top network coaccessible",""},
+ {"push (defined) <name>","adds a defined FSM to top of stack",""},
+ {"quit","exit foma",""},
+ {"read att <filename>","read a file in AT&T FSM format and add to top of stack","Short form: ratt"},
+ {"read cmatrix <filename>","read a confusion matrix and associate it with the network on top of the stack",""},
+ {"read prolog <filename>","reads prolog format file",""},
+ {"read lexc <filename>","read and compile lexc format file",""},
+ {"read spaced-text <filename>","compile space-separated words/word-pairs separated by newlines into a FST",""},
+ {"read text <filename>","compile a list of words separated by newlines into an automaton",""},
+ {"reverse net","reverses top FSM","Short form: rev\nSee .r\n"},
+ {"rotate stack","rotates stack",""},
+ {"save defined <filename>","save all defined networks to binary file","Short form: saved" },
+ {"save stack <filename>","save stack to binary file","Short form: ss" },
+ {"set <variable> <ON|OFF>","sets a global variable (see show variables)","" },
+ {"show variables","prints all variable/value pairs",""},
+ {"shuffle net","asynchronous product on top two FSMs on stack","See ∥ (or <>)\n"},
+ {"sigma net","Extracts the alphabet and creates a FSM that accepts all single symbols in it","See also: label net"},
+ {"source <file>","read and compile script file",""},
+ {"sort net","sorts arcs topologically on top FSM",""},
+ {"sort in","sorts input arcs by sigma numbers on top FSM",""},
+ {"sort out","sorts output arcs by sigma number on top FSM",""},
+ {"substitute defined X for Y","substitutes defined network X at all arcs containing Y ",""},
+ {"substitute symbol X for Y","substitutes all occurrences of Y in an arc with X",""},
+ {"system <cmd>","execute a system command","" },
+ {"test unambiguous","test if top FST is unambiguous","Short form: tunam\n"},
+ {"test equivalent","test if the top two FSMs are equivalent","Short form: equ\nNote: equivalence is undecidable for transducers in the general case. The result is reliable only for recognizers.\n"},
+ {"test functional","test if the top FST is functional (single-valued)","Short form: tfu\n"},
+ {"test identity","test if top FST represents identity relations only","Short form: tid\n"},
+ {"test lower-universal","test if lower side is Σ*","Short form: tlu\n"},
+ {"test upper-universal","test if upper side is Σ*","Short form: tuu\n"},
+ {"test non-null","test if top machine is not the empty language","Short form:tnn\n" },
+ {"test null","test if top machine is the empty language (∅)","Short form: tnu\n" },
+ {"test sequential","tests if top machine is sequential","Short form: tseq\n"},
+ {"test star-free","test if top FSM is star-free","Short form: tsf\n"},
+ {"turn stack","turns stack upside down","" },
+ {"twosided flag-diacritics","changes flags to always be identity pairs","Short form: tfd" },
+ {"undefine <name>","remove <name> from defined networks","See define\n"},
+ {"union net","union of top two FSMs","See ∪ (or |)\n"},
+ {"upper-side net","upper projection of top FSM","See ₁ (or .u)\n"},
+ {"view net","display top network (if supported)",""},
+ {"zero-plus net","Kleene star on top fsm","See *\n"},
+ {"variable compose-tristate","use the tristate composition algorithm","Default value: OFF\n"},
+ {"variable show-flags","show flag diacritics in `apply'","Default value: ON\n"},
+ {"variable obey-flags","obey flag diacritics in `apply'","Default value: ON\n"},
+ {"variable minimal","minimize resulting FSMs","Default value: ON\n"},
+ {"variable print-pairs","always print both sides when applying","Default value: OFF\n"},
+ {"variable print-space","print spaces between symbols","Default value: OFF\n"},
+ {"variable print-sigma","print the alphabet when printing network","Default value: ON\n"},
+ {"quit-on-fail","Abort operations when encountering errors","Default value: ON\n"},
+ {"variable recursive-define","Allow recursive definitions","Default value: OFF\n"},
+ {"variable verbose","Verbosity of interface","Default value: ON\n"},
+ {"variable hopcroft-min","ON = Hopcroft minimization, OFF = Brzozowski minimization","Default value: ON\n"},
+ {"variable med-limit","the limit on number of matches in apply med","Default value: 3\n"},
+ {"variable med-cutoff","the cost limit for terminating a search in apply med","Default value: 3\n"},
+ {"variable att-epsilon","the EPSILON symbol when reading/writing AT&T files","Default value: @0@\n"},
+ {"variable lexc-align","Forces X:0 X:X of 0:X alignment of lexicon entry symbols","Default value: OFF\n"},
+ {"write prolog (> filename)","writes top network to prolog format file/stdout","Short form: wpl"},
+ {"write att (> <filename>)","writes top network to AT&T format file/stdout","Short form: watt"},
+ {"re operator: (∀<var name>)(F)","universal quantification","Example: $.A is equivalent to:\n(∃x)(x ∈ A ∧ (∀y)(¬(y ∈ A ∧ ¬(x = y))))"},
+ {"re operator: (∃<var name>)(F)","existential quantification","Example: $.A is equivalent to:\n(∃x)(x ∈ A ∧ ¬(∃y)(y ∈ A ∧ ¬(x = y)))"},
+ {"logic re operator: ∈","`in' predicate for logical formulae",""},
+ {"logic re operator: S(t1,t2)","successor-of predicate for logical formulae",""},
+ {"logic re operator: ≤","less-than or equal-to","Refers to position of quantified substring\n" },
+ {"logic re operator: ≥","more-than or equal-to","Refers to position of quantified substring\n" },
+ {"logic re operator: ≺","precedes","Refers to position of quantified substring\n"},
+ {"logic re operator: ≻","follows","Refers to position of quantified substring\n"},
+ {"logic re operator: ∧","conjunction","Operationally equivalent to ∩\n"},
+ {"logic re operator: ∨","disjunction","Operationally equivalent to ∪\n"},
+ {"logic re operator: →","implication","A → B is equivalent to ¬A ∨ B "},
+ {"logic re operator: ↔","biconditional","A ↔ B is equivalent to (¬A ∨ B) ∧ (¬B ∨ A)"},
+ {"re operator: ∘ (or .o.) ","compose","A .o. B is the composition of transducers/recognizers A and B\nThe composition algorithm can be controlled with the variable\ncompose-tristate. The default algorithm is a `bistate' composition that eliminates redundant paths but may fail to find the shortest path.\n"},
+ {"re operator: × (or .x.) ","cross-product","A × B (where A and B are recognizers, not transducers\nyields the cross-product of A and B.\n"},
+ {"re operator: .O. ","`lenient' composition","Lenient composition as defined in Karttunen(1998) A .O. B = [A ∘ B] .P. B\n"},
+ {"re operator: ∥ (or <>) ","shuffle (asynchronous product)","A ∥ B yields the asynchronous (or shuffle) product of FSM A and B.\n" },
+ {"re operator: => ","context restriction, e.g. A => B _ C, D _ E","A => B _ C yields the language where every instance of a substring drawn from A is surrounded by B and C. Multiple contexts can be specified if separated by commas, e.g.: A => B _ C, D _ E"},
+ {"re operator: ->, <-, <->, etc.","replacement operators","If LHS is a transducer, no RHS is needed in rule."},
+ {"re operator: @->, @>, etc.","directed replacement operators",""},
+ {"re operator: (->), (@->), etc. ","optional replacements","Optional replacement operators variants. Note that the directional modes leftmost/rightmost/longest/shortest are not affected by optionality, i.e. only replacement is optional, not mode. Hence A (@->) B is not in general equivalent to the parallel rule A @-> B, A -> ... "},
+ {"re operator: ||,\\/,\\\\,// ","replacement direction specifiers","Rewrite rules direction specifier meaning is:\nA -> B || C _ D (replace if C and D match on upper side)\nA -> B // C _ D (replace if C matches of lower side and D matches on upper side)\nA -> B \\\\ C _ D (replace if C matches on upper side and D matches on lower side)\nA -> B \\/ C _ D (replace if C and D match on lower side)\n"},
+ {"re operator: _ ","replacement or restriction context specifier",""},
+ {"re operator: ,,","parallel context replacement operator","Separates parallel rules, e.g.:\nA -> B , C @-> D || E _ F ,, G -> H \\/ I _ J\n"},
+ {"re operator: ,","parallel replacement operator","Separates rules and contexts. Example: A -> B, C <- D || E _ F"},
+ {"re operator: [.<r.e.>.]","single-epsilon control in replacement LHS, e.g. [..] -> x","If the LHS contains the empty string, as does [.a*.] -> x, the rule yields a transducer where the empty string is assumed to occur exactly once between each symbol."},
+ {"re operator: ...","markup replacement control (e.g. A -> B ... C || D _ E)","A -> B ... C yields a replacement transducer where the center A is left untouched and B and C inserted around A." },
+ {"re operator: ","concatenation","Binary operator: A B\nConcatenation is performed implicitly according to its precedence level without overt specification\n"},
+ {"re operator: ∪ (or |) ","union","Binary operator: A|B"},
+ {"re operator: ∩ (or &) ","intersection","Binary operator: A&B" },
+ {"re operator: - ","set minus","Binary operator A-B"},
+ {"re operator: .P.","priority union (upper)","Binary operator A .P. B\nEquivalent to: A .P. B = A ∪ [¬[A₁] ∘ B]\n" },
+ {"re operator: .p.","priority union (lower)","Binary operator A .p. B\nEquivalent to: A .p. B = A ∪ [¬[A₂] ∘ B]" },
+ {"re operator: <","precedes","Binary operator A < B\nYields the language where no instance of A follows an instance of B."},
+ {"re operator: >","follows","Binary operator A > B\nYields the language where no instance of A precedes an instance of B."},
+ {"re operator: /","ignore","Binary operator A/B\nYield the language/transducer where arbitrary sequences of strings/mappings from B are interspersed in A. For single-symbol languages B, A/B = A ∥ B*"},
+ {"re operator: ./.","ignore except at edges","Yields the language where arbitrary sequences from B are interspersed in A, except as the very first and very last symbol."},
+ {"re operator: \\\\\\","left quotient","Binary operator: A\\\\\\B\nInformally: the set of suffixes one can add to A to get strings in B\n"},
+ {"re operator: ///","right quotient","Binary operator A///B\nInformally: the set of prefixes one can add to B to get a string in A\n"},
+ {"re operator: /\\/","interleaving quotient","Binary operator A/\\/B\nInformally: the set of strings you can interdigitate (non-continuously) to B to get strings in A\n"},
+ {"re operator: ¬ (or ~) ","complement","Unary operator ~A, equivalent to Σ* - A\n"},
+ {"re operator: $","contains a factor of","Unary operator $A\nEquivalent to: Σ* A Σ*\n"},
+ {"re operator: $.","contains exactly one factor of","Unary operator $.A\nYields the language that contains exactly one factor from A.\nExample: if A = [a b|b a], $.A contains strings ab, ba, abb, bba, but not abab, baba, aba, bab, etc.\n"},
+ {"re operator: $?","contains maximally one factor of","Unary operator: $?A, yields the language that contains zero or one factors from A. See also $.A."},
+ {"re operator: +","Kleene plus","Unary operator A+\n"},
+ {"re operator: *","Kleene star","Unary operator A*\n" },
+ {"re operator: ^n ^<n ^>n ^{m,n}","m, n-ary concatenations","A^n: A concatenated with itself exactly n times\nA^<n: A concatenated with itself less than n times\nA^>n: A concatenated with itself more than n times\nA^{m,n}: A concatenated with itself between m and n times\n"},
+ {"re operator: ₁ (or .1 or .u)","upper projection","Unary operator A.u\n"},
+ {"re operator: ₂ (or .2 or .l)","lower projection","Unary operator A.l\n"},
+ {"re operator: ⁻¹ (or .i)","inverse of transducer","Unary operator A.i\n"},
+ {"re operator: .f","eliminate all flags","Unary operator A.f: eliminates all flag diacritics in A"},
+ {"re operator: .r","reverse of FSM","Unary operator A.r\n"},
+ {"re operator: :","cross-product","Binary operator A:B, see also A × B\n"},
+ {"re operator: \\","term complement (\\x = [Σ-x])","Unary operator \\A\nSingle symbols not in A. Equivalent to [Σ-A]\n"},
+ {"re operator: `","substitution/homomorphism","Ternary operator `[A,B,C] Replace instances of symbol B with symbol C in language A. Also removes the substituted symbol from the alphabet.\n"},
+ {"re operator: { ... }","concatenate symbols","Single-symbol-concatenation\nExample: {abcd} is equivalent to a b c d\n"},
+ {"re operator: (A)","optionality","Equivalent to A | ε\nNote: parentheses inside logical formulas function as grouping, see ∀,∃\n"},
+ {"re operator: @\"filename\"","read saved network from file","Note: loads networks stored with e.g. \"save stack\" but if file contains more than one network, only the first one is used in the regular expression. See also \"load stack\" and \"load defined\"\n"},
+ {"special symbol: Σ (or ?)","`any' symbol in r.e.",""},
+ {"special symbol: ε (or 0, [])","epsilon symbol in r.e.",""},
+ {"special symbol: ∅","the empty language symbol in r.e.",""},
+ {"special symbol: .#.","word boundary symbol in replacements, restrictions","Signifies both end and beginning of word/string\nExample: A => B _ .#. (allow A only between B and end-of-string)\nExample: A -> B || .#. _ C (replace A with B if it occurs in the beginning of a word and is followed by C)\n"},
+ {"operator precedence: ","see: `help precedence'","\\ `\n:\n+ * ^ ₁ ₂ ⁻¹ .f .r\n¬ $ $. $?\n(concatenation)\n> <\n∪ ∩ - .P. .p.\n=> -> (->) @-> etc.\n∥\n× ∘ .O.\nNote: compatibility variants (i.e. | = ∪ etc.) are not listed."},
+
+ {NULL,NULL,NULL}
+};
+
+void iface_help() {
+ struct global_help *gh;
+ int i, maxlen;
+
+ for (maxlen = 0, gh = global_help; gh->name != NULL; gh++) {
+ maxlen = maxlen < utf8strlen(gh->name) ? utf8strlen(gh->name) : maxlen;
+ }
+ for (gh = global_help; gh->name != NULL; gh++) {
+ printf("%s",gh->name);
+ for (i = maxlen - utf8strlen(gh->name); i>=0; i--) {
+ printf("%s"," ");
+ }
+ printf("%s\n",gh->help);
+ }
+}
+
+void iface_ambiguous_upper() {
+ if (iface_stack_check(1))
+ stack_add(fsm_extract_ambiguous_domain(stack_pop()));
+}
+
+void iface_apropos(char *s) {
+ struct global_help *gh;
+ int i, maxlen;
+
+ for (maxlen = 0, gh = global_help; gh->name != NULL; gh++) {
+ if (strstr(gh->name,s) != NULL || strstr(gh->help,s) != NULL) {
+ maxlen = maxlen < utf8strlen(gh->name) ? utf8strlen(gh->name) : maxlen;
+ }
+ }
+ for (gh = global_help; gh->name != NULL; gh++) {
+ if (strstr(gh->name,s) != NULL || strstr(gh->help,s) != NULL) {
+ printf("%s",gh->name);
+ for (i = maxlen - utf8strlen(gh->name); i>=0; i--) {
+ printf("%s"," ");
+ }
+ printf("%s\n",gh->help);
+ }
+ }
+}
+
+void iface_help_search(char *s) {
+ struct global_help *gh;
+
+ for (gh = global_help; gh->name != NULL; gh++) {
+ if (strstr(gh->name,s) != NULL || strstr(gh->help,s) != NULL) {
+ printf("##\n");
+ printf("%-32.32s%s\n%s\n",gh->name,gh->help,gh->longhelp);
+ }
+ }
+}
+
+void iface_print_bool(int value) {
+ printf("%i (1 = TRUE, 0 = FALSE)\n",value);
+}
+
+void iface_warranty() {
+ printf("%s",warranty);
+}
+
+void iface_apply_set_params(struct apply_handle *h) {
+ apply_set_print_space(h, g_print_space);
+ apply_set_print_pairs(h, g_print_pairs);
+ apply_set_show_flags(h, g_show_flags);
+ apply_set_obey_flags(h, g_obey_flags);
+}
+
+void iface_apply_med(char *word) {
+ char *result;
+ struct apply_med_handle *amedh;
+ if (!iface_stack_check(1)) {
+ return;
+ }
+ amedh = stack_get_med_ah();
+
+ apply_med_set_heap_max(amedh,4194304+1);
+ apply_med_set_med_limit(amedh,g_med_limit);
+ apply_med_set_med_cutoff(amedh,g_med_cutoff);
+
+ result = apply_med(amedh, word);
+ if (result == NULL) {
+ printf("???\n");
+ return;
+ } else {
+ printf("%s\n",result);
+ printf("%s\n", apply_med_get_instring(amedh));
+ printf("Cost[f]: %i\n\n", apply_med_get_cost(amedh));
+ }
+ while ((result = apply_med(amedh,NULL)) != NULL) {
+ printf("%s\n",result);
+ printf("%s\n", apply_med_get_instring(amedh));
+ printf("Cost[f]: %i\n\n", apply_med_get_cost(amedh));
+ }
+}
+
+int iface_apply_file(char *infilename, char *outfilename, int direction) {
+ char *result, inword[LINE_LIMIT];
+ struct apply_handle *ah;
+ FILE *OUTFILE, *INFILE;
+
+ if (direction != AP_D && direction != AP_U) {
+ perror("Invalid direction in iface_apply_file().\n");
+ return 1;
+ }
+ if (!iface_stack_check(1)) { return 0; }
+ INFILE = fopen(infilename, "r");
+ if (INFILE == NULL) {
+ fprintf(stderr, "%s: ", infilename);
+ perror("Error opening file");
+ return 1;
+ }
+
+ if (outfilename == NULL) {
+ OUTFILE = stdout;
+ } else {
+ OUTFILE = fopen(outfilename, "w");
+ printf("Writing output to file %s.\n", outfilename);
+ if (OUTFILE == NULL) {
+ fprintf(stderr, "%s: ", outfilename);
+ perror("Error opening output file.");
+ return 1;
+ }
+ }
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ while ((fgets(inword,LINE_LIMIT,INFILE)) != NULL) {
+ if (inword[strlen(inword)-1] == '\n') {
+ inword[strlen(inword)-1] = '\0';
+ }
+
+ fprintf(OUTFILE,"\n%s\n", inword);
+ if (direction == AP_D)
+ result = apply_down(ah,inword);
+ else
+ result = apply_up(ah,inword);
+
+ if (result == NULL) {
+ fprintf(OUTFILE,"???\n");
+ continue;
+ } else {
+ fprintf(OUTFILE,"%s\n",result);
+ }
+ for (;;) {
+ if (direction == AP_D)
+ result = apply_down(ah,NULL);
+ if (direction == AP_U)
+ result = apply_up(ah,NULL);
+ if (result == NULL)
+ break;
+ fprintf(OUTFILE,"%s\n", result);
+ }
+ }
+ if (outfilename != NULL)
+ fclose(OUTFILE);
+ return 0;
+}
+
+void iface_apply_down(char *word) {
+ int i;
+ char *result;
+ struct apply_handle *ah;
+ if (!iface_stack_check(1)) {
+ return;
+ }
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ result = apply_down(ah, word);
+ if (result == NULL) {
+ printf("???\n");
+ return;
+ } else {
+ printf("%s\n",result);
+ }
+ for (i = g_list_limit; i > 0; i--) {
+ result = apply_down(ah, NULL);
+ if (result == NULL)
+ break;
+ printf("%s\n",result);
+ }
+}
+
+void iface_apply_up(char *word) {
+ int i;
+ char *result;
+ struct apply_handle *ah;
+ if (!iface_stack_check(1)) {
+ return;
+ }
+ ah = stack_get_ah();
+
+ iface_apply_set_params(ah);
+ result = apply_up(ah, word);
+
+ if (result == NULL) {
+ printf("???\n");
+ return;
+ } else {
+ printf("%s\n",result);
+ }
+ for (i = g_list_limit; i > 0; i--) {
+ result = apply_up(ah, NULL);
+ if (result == NULL)
+ break;
+ printf("%s\n",result);
+ }
+}
+
+void iface_close() {
+ if (iface_stack_check(1)) {
+ stack_add(fsm_topsort(fsm_minimize(fsm_close_sigma(stack_pop(),0))));
+ }
+}
+
+void iface_compact() {
+ if (iface_stack_check(1)) {
+ fsm_compact(stack_find_top()->fsm);
+ sigma_sort(stack_find_top()->fsm);
+ stack_add(fsm_topsort(fsm_minimize(stack_pop())));
+ }
+}
+
+void iface_complete() {
+ if (iface_stack_check(1))
+ stack_add(fsm_complete(stack_pop()));
+}
+
+
+void iface_compose() {
+ struct fsm *one, *two;
+ if (iface_stack_check(2)) {
+ while (stack_size()>1) {
+ one = stack_pop();
+ two = stack_pop();
+ stack_add(fsm_topsort(fsm_minimize(fsm_compose(one,two))));
+ }
+ }
+}
+
+void iface_conc() {
+ struct fsm *one, *two;
+ if (iface_stack_check(2)) {
+ while (stack_size()>1) {
+ printf("dd");
+ one = stack_pop();
+ two = stack_pop();
+ stack_add(fsm_topsort(fsm_minimize(fsm_concat(one,two))));
+ }
+ }
+}
+
+void iface_crossproduct() {
+ struct fsm *one, *two;
+ if (iface_stack_check(2)) {
+ one = stack_pop();
+ two = stack_pop();
+ stack_add(fsm_topsort(fsm_minimize(fsm_cross_product(one,two))));
+ }
+}
+void iface_determinize() {
+ if (iface_stack_check(1))
+ stack_add(fsm_determinize(stack_pop()));
+}
+
+void iface_eliminate_flags() {
+ if (iface_stack_check(1))
+ stack_add(flag_eliminate(stack_pop(), NULL));
+}
+
+void iface_extract_ambiguous() {
+ if (iface_stack_check(1))
+ stack_add(fsm_extract_ambiguous(stack_pop()));
+}
+
+void iface_extract_unambiguous() {
+ if (iface_stack_check(1))
+ stack_add(fsm_extract_unambiguous(stack_pop()));
+}
+
+int iface_extract_number(char *s) {
+ int i;
+ for (i=0; *(s+i) != '\0' && ((unsigned char) *(s+i) < '0' || (unsigned char) *(s+i) > '9'); i++) { }
+ return(atoi(s+i));
+}
+
+void iface_eliminate_flag(char *name) {
+ if (iface_stack_check(1))
+ stack_add(flag_eliminate(stack_pop(), name));
+}
+
+void iface_factorize() {
+ if (iface_stack_check(1))
+ stack_add(fsm_bimachine(stack_pop()));
+}
+
+void iface_sequentialize() {
+ if (iface_stack_check(1))
+ stack_add(fsm_sequentialize(stack_pop()));
+}
+
+void iface_ignore() {
+ struct fsm *one, *two;
+ if (iface_stack_check(2)) {
+ one = stack_pop();
+ two = stack_pop();
+ stack_add(fsm_topsort(fsm_minimize(fsm_ignore(one,two,OP_IGNORE_ALL))));
+ }
+}
+
+void iface_intersect() {
+ if (iface_stack_check(2)) {
+ while (stack_size()>1)
+ stack_add(fsm_topsort(fsm_minimize(fsm_intersect(stack_pop(),stack_pop()))));
+ }
+}
+
+void iface_invert() {
+ if (iface_stack_check(1))
+ stack_add(fsm_invert(stack_pop()));
+}
+
+void iface_label_net() {
+ if (iface_stack_check(1))
+ stack_add(fsm_sigma_pairs_net(stack_pop()));
+}
+
+void iface_letter_machine() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_minimize(fsm_letter_machine(stack_pop()))));
+}
+
+void iface_load_defined(char *filename) {
+ load_defined(g_defines, filename);
+}
+
+void iface_load_stack(char *filename) {
+ struct fsm *net;
+ fsm_read_binary_handle fsrh;
+
+ if ((fsrh = fsm_read_binary_file_multiple_init(filename)) == NULL) {
+ fprintf(stderr, "%s: ", filename);
+ perror("File error");
+ return;
+ }
+ while ((net = fsm_read_binary_file_multiple(fsrh)) != NULL)
+ stack_add(net);
+ return;
+}
+
+void iface_lower_side() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_minimize(fsm_lower(stack_pop()))));
+}
+
+void iface_minimize() {
+ int store_minimal_var;
+ if (iface_stack_check(1)) {
+ store_minimal_var = g_minimal;
+ g_minimal = 1;
+ stack_add(fsm_topsort(fsm_minimize(stack_pop())));
+ g_minimal = store_minimal_var;
+ }
+}
+
+void iface_one_plus() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_minimize(fsm_kleene_plus(stack_pop()))));
+}
+
+void iface_pop() {
+ struct fsm *net;
+ if (stack_size() < 1)
+ printf("Stack is empty.\n");
+ else {
+ net = stack_pop();
+ fsm_destroy(net);
+ }
+}
+
+void iface_lower_words(int limit) {
+ char *result;
+ struct apply_handle *ah;
+ int i;
+ if (!iface_stack_check(1)) {
+ return;
+ }
+ limit = (limit == -1) ? g_list_limit : limit;
+ if (iface_stack_check(1)) {
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ for (i = limit; i > 0; i--) {
+ result = apply_lower_words(ah);
+ if (result == NULL)
+ break;
+ printf("%s\n",result);
+ }
+ apply_reset_enumerator(ah);
+ }
+}
+
+void iface_name_net(char *name) {
+ if (iface_stack_check(1)) {
+ strncpy(stack_find_top()->fsm->name, name, 40);
+ iface_print_name();
+ }
+}
+void iface_negate() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_minimize(fsm_complement(stack_pop()))));
+}
+
+void iface_print_dot(char *filename) {
+ if (iface_stack_check(1)) {
+ if (filename != NULL)
+ printf("Writing dot file to %s.\n",filename);
+ print_dot(stack_find_top()->fsm, filename);
+ }
+}
+
+void iface_print_net(char *netname, char *filename) {
+ struct fsm *net;
+ if (netname != NULL) {
+ if ((net = find_defined(g_defines, netname)) == NULL) {
+ printf("No defined network %s.\n", netname);
+ return;
+ }
+ print_net(net, filename);
+ } else {
+ if (iface_stack_check(1))
+ print_net(stack_find_top()->fsm, filename);
+ }
+}
+
+void iface_print_cmatrix_att(char *filename) {
+ FILE *outfile;
+ if (iface_stack_check(1)) {
+ if (stack_find_top()->fsm->medlookup == NULL || stack_find_top()->fsm->medlookup->confusion_matrix == NULL) {
+ printf("No confusion matrix defined.\n");
+ } else {
+ if (filename == NULL) {
+ outfile = stdout;
+ } else {
+ outfile = fopen(filename,"w");
+ printf("Writing confusion matrix to file '%s'.\n", filename);
+ }
+ cmatrix_print_att(stack_find_top()->fsm, outfile);
+ }
+ }
+}
+
+void iface_print_cmatrix() {
+ if (iface_stack_check(1)) {
+ if (stack_find_top()->fsm->medlookup == NULL || stack_find_top()->fsm->medlookup->confusion_matrix == NULL) {
+ printf("No confusion matrix defined.\n");
+ } else {
+ cmatrix_print(stack_find_top()->fsm);
+ }
+ }
+}
+
+void iface_print_defined() {
+ struct defined_networks *defined;
+ struct defined_functions *defined_f;
+ if (g_defines == NULL) {
+ printf("No defined symbols.\n");
+ }
+ for (defined = g_defines; defined != NULL; defined = defined->next) {
+ if (defined->name != NULL) {
+ printf("%s\t",defined->name);
+ print_stats(defined->net);
+ }
+ }
+ for (defined_f = g_defines_f; defined_f != NULL; defined_f = defined_f->next) {
+ if (defined_f->name != NULL) {
+ printf("%s@%i)\t",defined_f->name,defined_f->numargs);
+ printf("%s\n",defined_f->regex);
+ }
+ }
+}
+
+void iface_print_name() {
+ if (iface_stack_check(1))
+ printf("%s\n",stack_find_top()->fsm->name);
+}
+
+void iface_quit() {
+ struct fsm *net;
+ remove_defined(g_defines, NULL);
+ while (!(stack_isempty())) {
+ net = stack_pop();
+ fsm_destroy(net);
+ }
+ exit(0);
+}
+
+void iface_random_lower(int limit) {
+ iface_apply_random(&apply_random_lower, limit);
+}
+
+void iface_random_upper(int limit) {
+ iface_apply_random(&apply_random_upper, limit);
+}
+
+void iface_random_words(int limit) {
+ iface_apply_random(&apply_random_words, limit);
+}
+
+void iface_apply_random(char *(*applyer)(), int limit) {
+ char *result;
+ struct apply_handle *ah;
+ int i;
+ struct apply_results {
+ char *string;
+ int count;
+ } *results, *tempresults;
+
+ limit = (limit == -1) ? g_list_random_limit : limit;
+ if (iface_stack_check(1)) {
+ results = xxcalloc(limit, sizeof(struct apply_results));
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ for (i = limit; i > 0; i--) {
+ result = NULL;
+ result = applyer(ah);
+ if (result != NULL) {
+ for (tempresults = results; tempresults - results < limit; tempresults++) {
+ if (tempresults->string == NULL) {
+ tempresults->string = strdup(result);
+ tempresults->count = 1;
+ break;
+ }
+ else if (strcmp(tempresults->string, result) == 0) {
+ tempresults->count++;
+ break;
+ }
+ }
+ }
+ }
+ for (tempresults = results; tempresults - results < limit; tempresults++) {
+ if (tempresults->string != NULL) {
+ printf("[%i] %s\n", tempresults->count, tempresults->string);
+ xxfree(tempresults->string);
+ }
+ }
+ xxfree(results);
+ apply_reset_enumerator(ah);
+ }
+}
+
+void iface_print_sigma() {
+ if (iface_stack_check(1))
+ print_sigma(stack_find_top()->fsm->sigma,stdout);
+}
+void iface_print_stats() {
+ if (iface_stack_check(1))
+ print_stats(stack_find_top()->fsm);
+}
+
+void iface_print_shortest_string() {
+ /* L - ?+ [[L .o. [?:"@TMP@"]*].l .o. "@TMP@":?*].l; */
+ struct fsm *Result, *ResultU, *ResultL, *one, *onel, *oneu;
+ struct apply_handle *ah;
+ char *word;
+ if (iface_stack_check(1)) {
+ one = fsm_copy(stack_find_top()->fsm);
+ /* L - ?+ [[L .o. [?:"@TMP@"]*].l .o. "@TMP@":?*].l; */
+ if (stack_find_top()->fsm->arity == 1) {
+ Result = fsm_minimize(fsm_minus(fsm_copy(one),fsm_concat(fsm_kleene_plus(fsm_identity()),fsm_lower(fsm_compose(fsm_lower(fsm_compose(fsm_copy(one),fsm_kleene_star(fsm_cross_product(fsm_identity(),fsm_symbol("@TMP@"))))),fsm_kleene_star(fsm_cross_product(fsm_symbol("@TMP@"),fsm_identity())))))));
+ ah = apply_init(Result);
+ word = apply_words(ah);
+ if (word != NULL) printf("%s\n",word);
+ apply_clear(ah);
+ } else {
+ onel = fsm_lower(fsm_copy(one));
+ oneu = fsm_upper(one);
+ ResultU = fsm_minimize(fsm_minus(fsm_copy(oneu),fsm_concat(fsm_kleene_plus(fsm_identity()),fsm_lower(fsm_compose(fsm_lower(fsm_compose(fsm_copy(oneu),fsm_kleene_star(fsm_cross_product(fsm_identity(),fsm_symbol("@TMP@"))))),fsm_kleene_star(fsm_cross_product(fsm_symbol("@TMP@"),fsm_identity())))))));
+ ResultL = fsm_minimize(fsm_minus(fsm_copy(onel),fsm_concat(fsm_kleene_plus(fsm_identity()),fsm_lower(fsm_compose(fsm_lower(fsm_compose(fsm_copy(onel),fsm_kleene_star(fsm_cross_product(fsm_identity(),fsm_symbol("@TMP@"))))),fsm_kleene_star(fsm_cross_product(fsm_symbol("@TMP@"),fsm_identity())))))));
+ ah = apply_init(ResultU);
+ word = apply_words(ah);
+ if (word == NULL) word = "";
+ printf("Upper: %s\n",word);
+ apply_clear(ah);
+ ah = apply_init(ResultL);
+ word = apply_words(ah);
+ if (word == NULL) word = "";
+ printf("Lower: %s\n",word);
+ apply_clear(ah);
+ }
+ }
+}
+
+void iface_print_shortest_string_size() {
+ struct fsm *Result, *ResultU, *ResultL, *one, *onel, *oneu;
+ if (iface_stack_check(1)) {
+ one = fsm_copy(stack_find_top()->fsm);
+ /* [L .o. [?:a]*].l; */
+ if (stack_find_top()->fsm->arity == 1) {
+ Result = fsm_minimize(fsm_lower(fsm_compose(one,fsm_kleene_star(fsm_cross_product(fsm_identity(),fsm_symbol("a"))))));
+ printf("Shortest acyclic path length: %i\n",Result->statecount-1);
+
+ } else {
+ onel = fsm_lower(fsm_copy(one));
+ oneu = fsm_upper(one);
+ ResultU = fsm_minimize(fsm_lower(fsm_compose(oneu,fsm_kleene_star(fsm_cross_product(fsm_identity(),fsm_symbol("a"))))));
+ ResultL = fsm_minimize(fsm_lower(fsm_compose(onel,fsm_kleene_star(fsm_cross_product(fsm_identity(),fsm_symbol("a"))))));
+ printf("Shortest acyclic upper path length: %i\n",(ResultU->statecount)-1);
+ printf("Shortest acyclic lower path length: %i\n",(ResultL->statecount)-1);
+ }
+ }
+}
+
+int iface_read_att(char *filename) {
+ struct fsm *tempnet;
+ printf("Reading AT&T file: %s\n",filename);
+ tempnet = read_att(filename);
+ if (tempnet == NULL) {
+ fprintf(stderr, "%s: ", filename);
+ perror("Error opening file");
+ return 1;
+ } else {
+ stack_add(tempnet);
+ return 0;
+ }
+}
+
+int iface_read_prolog(char *filename) {
+ struct fsm *tempnet;
+ printf("Reading prolog: %s\n",filename);
+ tempnet = fsm_read_prolog(filename);
+ if (tempnet == NULL) {
+ fprintf(stderr, "%s: ", filename);
+ perror ("Error opening file");
+ return 1;
+ } else {
+ stack_add(tempnet);
+ return 0;
+ }
+}
+
+int iface_read_spaced_text(char *filename) {
+ struct fsm *net;
+ net = fsm_read_spaced_text_file(filename);
+ if (net == NULL) {
+ fprintf(stderr, "%s: ", filename);
+ perror("File error");
+ return 1;
+ }
+ stack_add(fsm_topsort(fsm_minimize(net)));
+ return 0;
+}
+
+int iface_read_text(char *filename) {
+ struct fsm *net;
+ net = fsm_read_text_file(filename);
+ if (net == NULL) {
+ fprintf(stderr, "%s: ", filename);
+ perror("File error");
+ return 1;
+ }
+ stack_add(fsm_topsort(fsm_minimize(net)));
+ return 0;
+}
+
+int iface_stack_check (int size) {
+ if (stack_size() < size) {
+ printf("Not enough networks on stack. Operation requires at least %i.\n",size);
+ return 0;
+ }
+ return 1;
+}
+
+void iface_substitute_symbol (char *original, char *substitute) {
+ if (iface_stack_check(1)) {
+ dequote_string(original);
+ dequote_string(substitute);
+ stack_add(fsm_topsort(fsm_minimize(fsm_substitute_symbol(stack_pop(), original, substitute))));
+ printf("Substituted '%s' for '%s'.\n", substitute, original);
+ }
+}
+
+void iface_substitute_defined (char *original, char *substitute) {
+ struct fsm *subnet;
+ struct fsm *newnet;
+ if (iface_stack_check(1)) {
+ dequote_string(original);
+ dequote_string(substitute);
+ if ((subnet = find_defined(g_defines, substitute)) == NULL) {
+ printf("No defined network '%s'.\n",substitute);
+ } else {
+ if (fsm_symbol_occurs(stack_find_top()->fsm, original, M_UPPER + M_LOWER) == 0) {
+ printf("Symbol '%s' does not occur.\n", original);
+ } else {
+ newnet = fsm_substitute_label(stack_find_top()->fsm, original, subnet);
+ stack_pop();
+ printf("Substituted network '%s' for '%s'.\n", substitute, original);
+ stack_add(fsm_topsort(fsm_minimize(newnet)));
+ }
+ }
+ }
+}
+
+void iface_upper_words(int limit) {
+ char *result;
+ struct apply_handle *ah;
+ int i;
+ limit = (limit == -1) ? g_list_limit : limit;
+ if (iface_stack_check(1)) {
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ for (i = limit; i > 0; i--) {
+ result = apply_upper_words(ah);
+ if (result == NULL)
+ break;
+ printf("%s\n",result);
+ }
+ apply_reset_enumerator(ah);
+ }
+}
+
+void iface_prune() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_coaccessible(stack_pop())));
+}
+void iface_reverse() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_determinize(fsm_reverse((stack_pop())))));
+}
+void iface_rotate() {
+ if (iface_stack_check(1))
+ stack_rotate();
+}
+void iface_save_defined(char *filename) {
+ save_defined(g_defines, filename);
+}
+
+void iface_save_stack(char *filename) {
+ gzFile outfile;
+ struct stack_entry *stack_ptr;
+
+ if (iface_stack_check(1)) {
+ if ((outfile = gzopen(filename, "wb")) == NULL) {
+ printf("Error opening file %s for writing.\n", filename);
+ return;
+ }
+ printf("Writing to file %s.\n", filename);
+ for (stack_ptr = stack_find_bottom(); stack_ptr->next != NULL; stack_ptr = stack_ptr->next) {
+ foma_net_print(stack_ptr->fsm, outfile);
+ }
+ gzclose(outfile);
+ return;
+ }
+}
+
+void iface_show_variables() {
+ int i;
+ for (i=0; global_vars[i].name != NULL; i++) {
+ if (global_vars[i].type == FVAR_BOOL) {
+ printf("%-17.17s: %s\n",global_vars[i].name, *((int *)(global_vars[i].ptr)) == 1 ? "ON" : "OFF");
+ }
+ if (global_vars[i].type == FVAR_INT) {
+ printf("%-17.17s: %i\n",global_vars[i].name, *((int *)(global_vars[i].ptr)));
+ }
+ if (global_vars[i].type == FVAR_STRING) {
+ printf("%-17.17s: %s\n",global_vars[i].name, *((char **)(global_vars[i].ptr)) );
+ }
+ }
+}
+void iface_show_variable(char *name) {
+ int i;
+ for (i=0; global_vars[i].name != NULL; i++) {
+ if (strncmp(name,global_vars[i].name,8) == 0) {
+ printf("%s = %s\n",global_vars[i].name, *((int *)(global_vars[i].ptr)) == 1 ? "ON" : "OFF");
+ return;
+ }
+ }
+ printf("*There is no global variable '%s'.\n",name);
+}
+
+void iface_set_variable(char *name, char *value) {
+ int i,j;
+ char *endptr;
+ for (i=0; global_vars[i].name != NULL; i++) {
+ if (strncmp(name,global_vars[i].name,8) == 0) {
+ if (global_vars[i].type == FVAR_BOOL) {
+ if ((strcmp(value,"ON") == 0) || (strcmp(value, "1") == 0)) {
+ j = 1;
+ } else if ((strcmp(value,"OFF") == 0) || (strcmp(value, "0") == 0)) {
+ j = 0;
+ } else {
+ printf("Invalid value '%s' for variable '%s'\n",value, global_vars[i].name);
+ return;
+ }
+ *((int *)(global_vars[i].ptr)) = j;
+ printf("variable %s = %s\n",global_vars[i].name, *((int *)(global_vars[i].ptr)) == 1 ? "ON" : "OFF");
+ return;
+ }
+ if (global_vars[i].type == FVAR_STRING) {
+ *((char **)(global_vars[i].ptr)) = xxstrdup(value);
+ printf("variable %s = %s\n",global_vars[i].name, value);
+ return;
+ }
+ if (global_vars[i].type == FVAR_INT) {
+ errno = 0;
+ j = strtol(value, &endptr, 10);
+ if ((errno != 0 || endptr == value) || j < 0) {
+ printf("invalid value %s for variable %s\n", value, global_vars[i].name);
+ return;
+ } else {
+ printf("variable %s = %i\n", global_vars[i].name, j);
+ *((int *)(global_vars[i].ptr)) = j;
+ return;
+ }
+ }
+ }
+ }
+ printf("*There is no global variable '%s'.\n",name);
+}
+
+void iface_shuffle() {
+ if (iface_stack_check(2))
+ while (stack_size()>1)
+ stack_add(fsm_minimize(fsm_shuffle(stack_pop(),stack_pop())));
+}
+
+void iface_sigma_net() {
+ if (iface_stack_check(1))
+ stack_add(fsm_sigma_net(stack_pop()));
+}
+
+void iface_sort_input() {
+ if (iface_stack_check(1)) {
+ fsm_sort_arcs(stack_find_top()->fsm,1);
+ }
+}
+
+void iface_sort_output() {
+ if (iface_stack_check(1)) {
+ fsm_sort_arcs(stack_find_top()->fsm,2);
+ }
+}
+
+void iface_sort() {
+ if (iface_stack_check(1)) {
+ sigma_sort(stack_find_top()->fsm);
+ stack_add(fsm_topsort(stack_pop()));
+ }
+}
+
+
+void iface_test_equivalent() {
+ struct fsm *one, *two;
+ if (iface_stack_check(2)) {
+ one = fsm_copy(stack_find_top()->fsm);
+ two = fsm_copy(stack_find_second()->fsm);
+ fsm_count(one);
+ fsm_count(two);
+
+ //if (one->arccount != two->arccount || one->statecount != two->statecount || one->finalcount != two->finalcount) {
+ //iface_print_bool(0);
+ //} else {
+ iface_print_bool(fsm_equivalent(one, two));
+ //iface_print_bool(fsm_isempty(fsm_union(fsm_minus(fsm_copy(one),fsm_copy(two)),fsm_minus(fsm_copy(two),fsm_copy(one)))));
+ //}
+ }
+}
+
+void iface_test_functional() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_isfunctional(stack_find_top()->fsm));
+}
+
+void iface_test_identity() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_isidentity(stack_find_top()->fsm));
+}
+
+void iface_test_nonnull() {
+ if (iface_stack_check(1))
+ iface_print_bool(!fsm_isempty(fsm_copy(stack_find_top()->fsm)));
+}
+
+void iface_test_null() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_isempty(fsm_copy(stack_find_top()->fsm)));
+}
+
+void iface_test_unambiguous() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_isunambiguous(stack_find_top()->fsm));
+}
+
+void iface_test_lower_universal() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_isempty(fsm_complement(fsm_lower(fsm_copy(stack_find_top()->fsm)))));
+}
+
+void iface_test_sequential() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_issequential(stack_find_top()->fsm));
+}
+
+void iface_test_upper_universal() {
+ if (iface_stack_check(1))
+ iface_print_bool(fsm_isempty(fsm_complement(fsm_upper(fsm_copy(stack_find_top()->fsm)))));
+}
+
+void iface_turn() {
+ if (iface_stack_check(1))
+ stack_rotate();
+}
+
+void iface_twosided_flags() {
+ if (iface_stack_check(1)) {
+ stack_add(flag_twosided(stack_pop()));
+ }
+}
+
+void iface_union() {
+ if (iface_stack_check(2))
+ while (stack_size()>1)
+ stack_add(fsm_minimize(fsm_union(stack_pop(),stack_pop())));
+}
+void iface_upper_side() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_minimize(fsm_upper(stack_pop()))));
+}
+
+void iface_view() {
+ if (iface_stack_check(1))
+ view_net(stack_find_top()->fsm);
+}
+
+void iface_words_file(char *filename, int type) {
+ /* type 0 (words), 1 (upper-words), 2 (lower-words) */
+ FILE *outfile;
+ char *result;
+ static char *(*applyer)() = &apply_words;
+ struct apply_handle *ah;
+
+ if (type == 1) {
+ applyer = &apply_upper_words;
+ }
+ if (type == 2) {
+ applyer = &apply_lower_words;
+ }
+ if (iface_stack_check(1)) {
+ if (stack_find_top()->fsm->pathcount == PATHCOUNT_CYCLIC) {
+ printf("FSM is cyclic: can't write all words to file.\n");
+ return;
+ }
+ printf("Writing to %s.\n",filename);
+ if ((outfile = fopen(filename, "w")) == NULL) {
+ perror("Error opening file");
+ return;
+ }
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ for (;;) {
+ result = applyer(ah);
+ if (result == NULL)
+ break;
+ fprintf(outfile,"%s\n",result);
+ }
+ apply_reset_enumerator(ah);
+ fclose(outfile);
+ }
+}
+
+void iface_words(int limit) {
+ char *result;
+ struct apply_handle *ah;
+ int i;
+ limit = (limit == -1) ? g_list_limit : limit;
+ if (iface_stack_check(1)) {
+ ah = stack_get_ah();
+ iface_apply_set_params(ah);
+ for (i = limit; i > 0; i--) {
+ result = apply_words(ah);
+ if (result == NULL)
+ break;
+ printf("%s\n",result);
+ }
+ apply_reset_enumerator(ah);
+ }
+}
+
+/* Splits string of upper:lower pairs with space separator into two strings */
+/* e.g. a:b c:d e 0:g => ace,bdeg */
+
+void iface_split_string(char *result, char *string) {
+ int i;
+ char space = '\001', epsilon = '\002', separator = '\003';
+ /* Simulate: SEPARATOR \SPACE+ @-> 0 .o. SPACE|SEPARATOR|EPSILON -> 0 */
+ /* to extract only upper side of string */
+ for (i = 0 ; ; ) {
+ zero:
+ if (result[i] == '\0') {
+ break;
+ } else if (result[i] == space || result[i] == epsilon) {
+ i++;
+ goto zero;
+ } else if (result[i] == separator) {
+ i++;
+ goto one;
+ } else {
+ strncat(string, result+i, 1);
+ i++;
+ goto zero;
+ }
+ one:
+ if (result[i] == '\0') {
+ break;
+ } else if (result[i] == space) {
+ i++;
+ goto zero;
+ } else {
+ i++;
+ goto one;
+ }
+ }
+}
+
+void iface_split_result(char *result, char **upper, char **lower) {
+ *upper = calloc(strlen(result), sizeof(char));
+ *lower = calloc(strlen(result), sizeof(char));
+ /* Split string into upper by filtering input side */
+ /* and lower by the same filter, but reversed */
+ iface_split_string(result, *upper);
+ xstrrev(result);
+ iface_split_string(result, *lower);
+ xstrrev(*lower);
+ xstrrev(result);
+}
+
+
+void iface_pairs_call(int limit, int random) {
+ char *result, *upper, *lower;
+ struct apply_handle *ah;
+ int i;
+ limit = (limit == -1) ? g_list_limit : limit;
+ if (iface_stack_check(1)) {
+ ah = stack_get_ah();
+ apply_set_show_flags(ah, g_show_flags);
+ apply_set_obey_flags(ah, g_obey_flags);
+ apply_set_space_symbol(ah, "\001");
+ apply_set_epsilon(ah, "\002");
+ apply_set_separator(ah, "\003");
+ for (i = limit; i > 0; i--) {
+ if (random == 1)
+ result = apply_random_words(ah);
+ else
+ result = apply_words(ah);
+ if (result == NULL)
+ break;
+ iface_split_result(result, &upper, &lower);
+ printf("%s\t%s\n",upper, lower);
+ xxfree(upper);
+ xxfree(lower);
+ }
+ apply_set_space_symbol(ah, " ");
+ apply_set_epsilon(ah, "0");
+ apply_set_separator(ah, ":");
+ apply_reset_enumerator(ah);
+ }
+}
+
+void iface_random_pairs(int limit) {
+ iface_pairs_call(limit, 1);
+}
+
+void iface_pairs(int limit) {
+ iface_pairs_call(limit, 0);
+}
+
+void iface_pairs_file(char *filename) {
+ FILE *outfile;
+ char *result, *upper, *lower;
+ struct apply_handle *ah;
+ if (iface_stack_check(1)) {
+ if (stack_find_top()->fsm->pathcount == PATHCOUNT_CYCLIC) {
+ printf("FSM is cyclic: can't write all pairs to file.\n");
+ return;
+ }
+ printf("Writing to %s.\n",filename);
+ if ((outfile = fopen(filename, "w")) == NULL) {
+ perror("Error opening file");
+ return;
+ }
+ ah = stack_get_ah();
+ apply_set_show_flags(ah, g_show_flags);
+ apply_set_obey_flags(ah, g_obey_flags);
+ apply_set_space_symbol(ah, "\001");
+ apply_set_epsilon(ah, "\002");
+ apply_set_separator(ah, "\003");
+ for (;;) {
+ result = apply_words(ah);
+ if (result == NULL)
+ break;
+ iface_split_result(result, &upper, &lower);
+ fprintf(outfile, "%s\t%s\n", upper, lower);
+ xxfree(upper);
+ xxfree(lower);
+ }
+ apply_set_space_symbol(ah, " ");
+ apply_set_epsilon(ah, "0");
+ apply_set_separator(ah, ":");
+ apply_reset_enumerator(ah);
+ fclose(outfile);
+ }
+}
+
+int iface_write_att(char *filename) {
+ FILE *outfile;
+ struct fsm *net;
+ if (!iface_stack_check(1)) {
+ return 1;
+ }
+ net = stack_find_top()->fsm;
+ if (filename == NULL) {
+ outfile = stdout;
+ } else {
+ printf("Writing AT&T file: %s\n",filename);
+ outfile = fopen(filename, "w");
+ if(outfile == NULL) {
+ fprintf(stderr, "%s: ", filename);
+ perror("File error opening.");
+ return 1;
+ }
+ }
+ net_print_att(net, outfile);
+ if (filename != NULL)
+ fclose(outfile);
+ return 0;
+}
+
+void iface_write_prolog(char *filename) {
+ if (iface_stack_check(1))
+ foma_write_prolog(stack_find_top()->fsm, filename);
+}
+
+void iface_zero_plus() {
+ if (iface_stack_check(1))
+ stack_add(fsm_topsort(fsm_minimize(fsm_kleene_star(stack_pop()))));
+}
+
+static char *sigptr(struct sigma *sigma, int number) {
+ char *mystr;
+ if (number == EPSILON)
+ return "0";
+ if (number == UNKNOWN)
+ return "?";
+ if (number == IDENTITY)
+ return "@";
+
+ for (; sigma != NULL; sigma = sigma->next) {
+ if (sigma->number == number) {
+ if (strcmp(sigma->symbol,"0") == 0)
+ return("\"0\"");
+ if (strcmp(sigma->symbol,"?") == 0)
+ return("\"?\"");
+ if (strcmp(sigma->symbol,"\n") == 0)
+ return("\\n");
+ if (strcmp(sigma->symbol,"\r") == 0)
+ return("\\r");
+ return (sigma->symbol);
+ }
+ }
+ mystr = xxmalloc(sizeof(char)*40);
+ snprintf(mystr, 40, "NONE(%i)",number);
+ return(mystr);
+}
+
+static int print_net(struct fsm *net, char *filename) {
+ struct fsm_state *stateptr;
+ int previous_state = -1, i;
+ int *finals;
+ FILE *out;
+ if (filename == NULL) {
+ out = stdout;
+ } else {
+ if ((out = fopen(filename, "w")) == NULL) {
+ printf("Error writing to file %s. Using stdout.\n", filename);
+ out = stdout;
+ }
+ printf("Writing network to file %s.\n", filename);
+ }
+ fsm_count(net);
+ finals = xxmalloc(sizeof(int)*(net->statecount));
+ stateptr = net->states;
+
+ for (i=0; (stateptr+i)->state_no != -1; i++) {
+ if ((stateptr+i)->final_state == 1) {
+ *(finals+((stateptr+i)->state_no)) = 1;
+ } else {
+ *(finals+((stateptr+i)->state_no)) = 0;
+ }
+ if ((stateptr+i)->in != (stateptr+i)->out) {
+ net->arity = 2;
+ }
+ }
+ print_sigma(net->sigma, out);
+ fprintf(out,"Net: %s\n",net->name);
+ fprintf(out,"Flags: ");
+ if (net->is_deterministic == YES) { fprintf(out,"deterministic ");}
+ if (net->is_pruned == YES) { fprintf(out,"pruned ");}
+ if (net->is_minimized == YES) { fprintf(out,"minimized ");}
+ if (net->is_epsilon_free == YES) { fprintf(out,"epsilon_free ");}
+ if (net->is_loop_free) { fprintf(out,"loop_free "); }
+ if (net->arcs_sorted_in) { fprintf(out,"arcs_sorted_in "); }
+ if (net->arcs_sorted_out) { fprintf(out,"arcs_sorted_out "); }
+ fprintf(out,"\n");
+ fprintf(out,"Arity: %i\n", net->arity);
+ for (; stateptr->state_no != -1; stateptr++) {
+ if (stateptr->state_no != previous_state) {
+ if (stateptr->start_state) {
+ fprintf(out,"S");
+ }
+ if (stateptr->final_state) {
+ fprintf(out,"f");
+ }
+ if (stateptr->in==-1) {
+ fprintf(out,"s%i:\t(no arcs).\n",stateptr->state_no);
+ continue;
+ } else {
+ fprintf(out,"s%i:\t",stateptr->state_no);
+ }
+ }
+ previous_state = stateptr->state_no;
+ if (stateptr->in == stateptr->out) {
+ if (stateptr->in == IDENTITY) {
+ fprintf(out,"@ -> ");
+ } else if (stateptr->in == UNKNOWN) {
+ fprintf(out,"?:? -> ");
+ } else {
+ fprintf(out,"%s -> ",sigptr(net->sigma, stateptr->in));
+ }
+ } else {
+ fprintf(out,"<%s:%s> -> ",sigptr(net->sigma, stateptr->in),sigptr(net->sigma, stateptr->out));
+ }
+ if (*(finals+(stateptr->target)) == 1) {
+ fprintf(out,"f");
+ }
+ fprintf(out,"s%i",stateptr->target);
+ if ((stateptr+1)->state_no == stateptr->state_no) {
+ fprintf(out,", ");
+ } else {
+ fprintf(out,".\n");
+ }
+
+ }
+ if (filename != NULL) {
+ fclose(out);
+ }
+ xxfree(finals);
+ return 0;
+}
+
+void print_mem_size(struct fsm *net) {
+ char size[20];
+ struct sigma *sigma;
+ unsigned int s;
+ float sf;
+ s = 0;
+ for (sigma = net->sigma; sigma != NULL && sigma->number != -1; sigma = sigma->next) {
+ s += strlen(sigma->symbol)+1+sizeof(struct sigma);
+ }
+ s += sizeof(struct fsm);
+ s += sizeof(struct fsm_state) * net->linecount;
+ sf = s;
+ if (s < 1024) {
+ sprintf(size, "%i bytes. ", s);
+ } else if (s >= 1024 && s < 1048576) {
+ sprintf(size, "%.1f kB. ", sf/1024);
+ } else if (s >= 1048576 && s < 1073741824) {
+ sprintf(size, "%.1f MB. ", sf/1048576);
+ } else if (s >= 1073741824) {
+ sprintf(size, "%.1f GB. ", sf/1073741824);
+ }
+ fprintf(stdout, "%s", size);
+ fflush(stdout);
+}
+
+int print_stats(struct fsm *net) {
+ print_mem_size(net);
+ if (net->statecount == 1) { printf("1 state, "); } else { printf("%i states, ",net->statecount); }
+ if (net->arccount == 1) { printf("1 arc, "); } else { printf("%i arcs, ",net->arccount); }
+ if (net->pathcount == 1)
+ printf("1 path");
+ else if (net->pathcount == -1)
+ printf("Cyclic");
+ else if (net->pathcount == -2)
+ printf("more than %lld paths",LLONG_MAX);
+ else if (net->pathcount == -3)
+ printf("unknown number of paths");
+ else
+ printf("%lld paths",net->pathcount);
+ printf(".\n");
+ return 0;
+}
+
+static int print_sigma(struct sigma *sigma, FILE *out) {
+ int size;
+ fprintf (out,"Sigma:");
+ for (size = 0; sigma != NULL; sigma = sigma->next) {
+ if (sigma->number > 2) {
+ fprintf(out," %s",(sigma->symbol));
+ size++;
+ }
+ if (sigma->number == IDENTITY) {
+ fprintf(out," %s","@");
+ }
+ if (sigma->number == UNKNOWN) {
+ fprintf(out," %s","?");
+ }
+ }
+ fprintf(out,"\n");
+ fprintf(out,"Size: %i.\n",size);
+ return(1);
+}
+
+static int print_dot(struct fsm *net, char *filename) {
+ struct fsm_state *stateptr;
+ FILE *dotfile;
+ int i, j, linelen;
+ short *finals, *printed;
+
+ fsm_count(net);
+
+ finals = xxmalloc(sizeof(short)*net->statecount);
+ stateptr = net->states;
+
+ for (i=0; (stateptr+i)->state_no != -1; i++) {
+ if ((stateptr+i)->final_state == 1) {
+ *(finals+((stateptr+i)->state_no)) = 1;
+ } else {
+ *(finals+((stateptr+i)->state_no)) = 0;
+ }
+ }
+
+ if (filename != NULL) {
+ dotfile = fopen(filename,"w");
+ } else {
+ dotfile = stdout;
+ }
+
+ fprintf(dotfile,"digraph A {\nrankdir = LR;\n");
+ /* Go through states */
+ for (i=0; i < net->statecount; i++) {
+ if (*(finals+i)) {
+ fprintf(dotfile,"node [shape=doublecircle,style=filled] %i\n",i);
+ } else {
+ fprintf(dotfile,"node [shape=circle,style=filled] %i\n",i);
+ }
+ }
+
+ printed = xxcalloc(net->linecount,sizeof(printed));
+ /* Go through arcs */
+ for (i=0; (stateptr+i)->state_no != -1; i++) {
+ if ((stateptr+i)->target == -1 || printed[i] == 1)
+ continue;
+ fprintf(dotfile,"%i -> %i [label=\"", (stateptr+i)->state_no, (stateptr+i)->target);
+ linelen = 0;
+ for (j=i; (stateptr+j)->state_no == (stateptr+i)->state_no; j++) {
+ if (((stateptr+i)->target == ((stateptr+j)->target)) && printed[j] == 0) {
+ printed[j] = 1;
+
+ if (((stateptr+j)->in == ((stateptr+j)->out)) && (stateptr+j)->out != UNKNOWN ) {
+ fprintf(dotfile,"%s", escape_string(sigptr(net->sigma, (stateptr+j)->in),'"'));
+ linelen += strlen((sigptr(net->sigma, (stateptr+j)->in)));
+ } else {
+ fprintf(dotfile,"<%s:%s>", escape_string(sigptr(net->sigma, (stateptr+j)->in),'"'), escape_string(sigptr(net->sigma, (stateptr+j)->out),'"'));
+ linelen += strlen((sigptr(net->sigma, (stateptr+j)->in))) + strlen(sigptr(net->sigma, (stateptr+j)->out)) + 3;
+ }
+ if (linelen > 12) {
+ fprintf(dotfile, "\\n");
+ linelen = 0;
+ } else {
+ fprintf(dotfile, " ");
+ }
+ }
+ }
+ fprintf(dotfile,"\"];\n");
+ }
+
+
+ xxfree(finals);
+ xxfree(printed);
+ fprintf(dotfile, "}\n");
+ if (filename != NULL)
+ fclose(dotfile);
+ return(1);
+}
+
+static int view_net(struct fsm *net) {
+
+ char tmpstr[255];
+ char *dotname;
+#ifndef __APPLE__
+ char *pngname;
+#endif /* __APPLE__ */
+
+ dotname = strncpy(tmpstr,tempnam(NULL,"foma"), 250);
+ strcat(dotname, ".dot");
+ dotname = xxstrdup(tmpstr);
+ print_dot(net, dotname);
+
+#ifdef __APPLE__
+ sprintf(tmpstr,"/usr/bin/open -a Graphviz %s &",dotname);
+ if (system(tmpstr) == -1)
+ printf("Error opening viewer.\n");
+
+#endif /* __APPLE__ */
+
+#ifndef __APPLE__
+ pngname = xxstrdup(tempnam(NULL, "foma"));
+ sprintf(tmpstr,"dot -Tpng %s > %s ",dotname,pngname);
+ if (system(tmpstr) == -1)
+ printf("Error writing tempfile.\n");
+ sprintf(tmpstr,"/usr/bin/xdg-open %s 2>/dev/null &",pngname);
+ if (system(tmpstr) == -1)
+ printf("Error opening viewer.\n");
+ xxfree(pngname);
+#endif /* __APPLE__ */
+
+ xxfree(dotname);
+
+ return(1);
+}
diff --git a/int_stack.c b/int_stack.c
new file mode 100644
index 0000000..2d2e61a
--- /dev/null
+++ b/int_stack.c
@@ -0,0 +1,96 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include "foma.h"
+
+#define MAX_STACK 2097152
+#define MAX_PTR_STACK 2097152
+
+static int a[MAX_STACK];
+static int top = -1;
+
+static void *ptr_stack[MAX_PTR_STACK];
+static int ptr_stack_top = -1;
+
+int ptr_stack_isempty() {
+ return ptr_stack_top == -1;
+}
+
+void ptr_stack_clear() {
+ ptr_stack_top = -1;
+}
+
+void *ptr_stack_pop() {
+ return ptr_stack[ptr_stack_top--];
+}
+
+int ptr_stack_isfull() {
+ return (ptr_stack_top == (MAX_PTR_STACK - 1));
+}
+
+void ptr_stack_push(void *ptr) {
+ if (ptr_stack_isfull()) {
+ fprintf(stderr, "Pointer stack full!\n");
+ exit(1);
+ }
+ ptr_stack[++ptr_stack_top] = ptr;
+}
+
+
+int int_stack_isempty() {
+ return top == -1;
+}
+
+void int_stack_clear() {
+ top = -1;
+}
+
+int int_stack_find (int entry) {
+ int i;
+ if (int_stack_isempty()) {
+ return 0;
+ }
+ for(i = 0; i <= top ; i++) {
+ if (entry == a[i]) {
+ return 1;
+ }
+ }
+ return 0;
+}
+
+int int_stack_size () {
+ return (top + 1);
+}
+
+void int_stack_push(int c) {
+ if (int_stack_isfull()) {
+ fprintf(stderr, "Stack full!\n");
+ exit(1);
+ }
+ a[++top] = c;
+}
+
+
+int int_stack_pop() {
+ return a[top--];
+}
+
+int int_stack_isfull() {
+ return (top == (MAX_STACK - 1));
+}
diff --git a/interface.l b/interface.l
new file mode 100644
index 0000000..c2129b5
--- /dev/null
+++ b/interface.l
@@ -0,0 +1,655 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+%option noyywrap
+%option nounput
+%option noinput
+%option yylineno
+%top{
+#define YY_BUF_SIZE 1048576
+}
+%{
+#include <stdio.h>
+#include "foma.h"
+#define RE 0 /* regex */
+#define DE 1 /* define */
+
+char funcdef[16384];
+
+struct func_args {
+ char *arg;
+ int argno;
+ struct func_args *next;
+};
+
+struct func_args *func_args = NULL;
+
+static char *func_name;
+static char *tempstr = NULL;
+static char *lexcfilein;
+static char *cmatrixfilein;
+static int pmode;
+static int func_arg_no;
+static int olddef;
+static int applydir;
+static struct fsm *tempnet;
+extern int promptmode;
+extern int apply_direction;
+extern int g_list_limit;
+
+extern int my_yyparse(char *my_string, int lineno, struct defined_networks *defined_nets, struct defined_functions *defined_funcs);
+extern void my_cmatrixparse(struct fsm *net, char *my_string);
+extern struct fsm *current_parse;
+extern struct fsm *fsm_lexc_parse_string(char *string, int verbose);
+extern int interfacelex();
+extern struct fsm *current_parse;
+extern void lexc_trim(char *s);
+
+int input_is_file;
+
+int get_iface_lineno(void) {
+ return(yylineno);
+}
+
+void clear_func_args(void) {
+ struct func_args *fa, *fp;
+ for (fa = func_args; fa != NULL; ) {
+ xxfree(fa->arg);
+ fp = fa;
+ fa = fa->next;
+ xxfree(fp);
+ }
+ func_args = NULL;
+}
+
+void add_func_arg(char *s) {
+ struct func_args *fa;
+ fa = xxmalloc(sizeof(struct func_args));
+ fa->arg = xxstrdup(s);
+ fa->next = func_args;
+ fa->argno = func_arg_no;
+ func_arg_no++;
+ func_args = fa;
+}
+
+char *rep_func_arg(char *s) {
+ struct func_args *fa;
+ char *argstr;
+ for (fa = func_args; fa != NULL; fa = fa->next) {
+ if (strcmp(fa->arg,s) == 0) {
+ argstr = xxmalloc(sizeof(char)*20);
+ sprintf(argstr, "@ARGUMENT%02i@", fa->argno);
+ return(argstr);
+ }
+ }
+ return(strdup(s));
+}
+
+void my_interfaceparse(char *my_string) {
+
+ YY_BUFFER_STATE my_string_buffer;
+ my_string_buffer = interface_scan_string(my_string);
+ interfacelineno = 1;
+ func_args = NULL;
+ interfacelex();
+ //interface_delete_buffer(my_string_buffer);
+}
+
+%}
+
+NONRESERVED [0-9A-Za-z\?\'\=]|[\300-\301].|[\302]([\000-\377]{-}[\254])|[\303]([\000-\377]{-}[\227])|[\304-\315].|[\316]([\000-\377]{-}[\243\265])|[\317-\337].|[\340-\341]..|[\342][\000-\200].|[\342][201][\000-\377]{-}[\273]|[\342][\202][\000-\377]{-}[\201\202]|[\342][\203-\205][\000-\377][\342][\206]([\000-\377]{-}[\222\224])|[\342][\207].|[\342][\210]([\000-\377]{-}[\200\203\205\210\230\245\247\250\251\252])|[\342][\211]([\000-\377]{-}[\240\244\245\272\273])|[\342][\212-\377].|[\343-\3 [...]
+
+NONL [\001-\177]{-}[\012\015]|[\300-\337].|[\340-\357]..|[\360-\367]...
+ANY [\001-\177]|[\300-\337].|[\340-\357]..|[\360-\367]...
+
+NONE [\001-\177]{-}[\073\173\175\042\045]|[\300-\301].|[\302]([\000-\377]{-}[\254])|[\303]([\000-\377]{-}[\227])|[\304-\315].|[\316]([\000-\377]{-}[\243\265])|[\317-\337].|[\340-\341]..|[\342][\000-\205].|[\342][\206]([\000-\377]{-}[\222\224])|[\342][\207].|[\342][\210]([\000-\377]{-}[\200\203\205\210\230\245\247\250\251\252])|[\342][\211]([\000-\377]{-}[\240\244\245\272\273])|[\342][\212-\377].|[\343-\357]..|[\360-\367]...
+NONESCAPED [\001-\177]{-}[\073\173\175\042\045]|[\300-\337].|[\340-\357]..|[\360-\367]...
+INREGEX [\001-\177]{-}[\041\043\073\173\175\042\045]|[\300-\337].|[\340-\357]..|[\360-\367]...
+
+
+BRACED [{]([^}]|[\300-\337].|[\340-\357]..|[\360-\367]...)+[}]
+QUOTED [\042]([^"]|[\300-\337].|[\340-\357]..|[\360-\367]...)+["]
+SP [\040]|[\011]
+NOSP [^ \t];
+NOSPEQ [\001-\177]{-}[\040\041\043\075]|[\300-\337].|[\340-\357]..|[\360-\367]...
+
+%x REGEX DEFI DEF SOURCE APPLY_DOWN APPLY_FILE_IN APPLY_MED APPLY_UP APPLY_P ELIMINATE_FLAG UNDEFINE RPL RLEXC RCMATRIX READ_TEXT READ_SPACED_TEXT SHOW_VAR SET_VAR SET_VALUE SAVE_STACK SAVE_DEFINED LOAD_STACK LOAD_DEFINED IGNORELINE REGEXQ REGEXB PUSH NAME_NET ECHO SYSTEM FUNC_1 FUNC_2 FUNC_3 FUNC_4 APROPOS HELP PRINT_NET_FILE PRINT_NET_NAME PRINT_NET_NAME_FILE PRINT_NET_NAME_FILE2 PRINT_DOT_FILE PRINT_DOT_NAME SUBSTITUTE_SYMBOL SUBSTITUTE_SYMBOL_2 SUBSTITUTE_SYMBOL_3 SUBSTITUTE_DEFINED [...]
+
+%%
+%{
+ if (YY_START == APPLY_P && promptmode == PROMPT_MAIN)
+ BEGIN(INITIAL);
+%}
+
+^{SP}*(ambiguous{SP}+upper|ambiguous){SP}* { iface_ambiguous_upper(); }
+^{SP}*(apply{SP}+)?down{SP}*/[^ \t<] { BEGIN(APPLY_DOWN);}
+^{SP}*(apply{SP}+)?down{SP}* { if (iface_stack_check(1)) {promptmode = PROMPT_A; apply_direction = AP_D; BEGIN(APPLY_P);}}
+^{SP}*(apply{SP}+)?down{SP}*[ \t]*<[ ]* { applydir = AP_D; BEGIN(APPLY_FILE_IN);}
+^{SP}*(apply{SP}+)?med{SP}* { if (iface_stack_check(1)) {promptmode = PROMPT_A; apply_direction = AP_M; BEGIN(APPLY_P);}}
+^{SP}*(apply{SP}+)?med{SP}*/[^ \t] { BEGIN(APPLY_MED);}
+^{SP}*(apply{SP}+)?up{SP}*/[^ \t] { BEGIN(APPLY_UP);}
+^{SP}*(apply{SP}+)?up{SP}* { if (iface_stack_check(1)) {promptmode = PROMPT_A; apply_direction = AP_U; BEGIN(APPLY_P);}}
+^{SP}*(apply{SP}+)?up{SP}*[ \t]*<[ ]* { applydir = AP_U; BEGIN(APPLY_FILE_IN);}
+^{SP}*apr(o(p(os?)?)?)?{SP}+/[^ ] {BEGIN(APROPOS); }
+^{SP}*assert-stack{SP} {BEGIN(ASSERT_STACK);}
+^{SP}*clear({SP}+st(a(ck?)?)?)? { stack_clear();}
+^{SP}*close({SP}+si(g(ma?)?)?)? { iface_close();}
+^{SP}*comp(a(ct?)?){SP}+sig(ma?)? { iface_compact(); }
+^{SP}*compl(e(te?)?)?({SP}+net?)? { iface_complete();}
+^{SP}*compo(se?)?({SP}+net?)? { iface_compose();}
+^{SP}*conc(a(t(e(n(a(te?)?)?)?)?)?)?({SP}+net?)? {iface_conc();}
+^{SP}*cross(p(r(o(d(u(ct?)?)?)?)?)?)?({SP}+net?)? { iface_crossproduct();}
+^{SP}*de(f(i(ne?)?)?)?{SP}+ { pmode = DE; BEGIN(DEFI); }
+^{SP}*{NOSPEQ}+{SP}*= { pmode = DE; lexc_trim(interfacetext); tempstr = xxstrdup(interfacetext); BEGIN(REGEX); }
+^{SP}*det(e(r(m(i(n(i(ze?)?)?)?)?)?)?)?({SP}+net?)? { iface_determinize(); }
+^{SP}*echo{SP} { BEGIN(ECHO); }
+^{SP}*echo { printf("\n"); }
+^{SP}*eliminate{SP}+flags { iface_eliminate_flags(); }
+^{SP}*eliminate{SP}+flag{SP}+ { BEGIN(ELIMINATE_FLAG); }
+^{SP}*export{SP}+cmatrix { iface_print_cmatrix_att(NULL); }
+^{SP}*export{SP}+cmatrix{SP}*>?{SP}*/[^ \t] { BEGIN(EXCMATRIX); }
+^{SP}*(extract{SP}+ambiguous|examb){SP}* { iface_extract_ambiguous(); }
+^{SP}*(extract{SP}+unambiguous|exunamb){SP}* { iface_extract_unambiguous(); }
+^{SP}*(fac|factorize){SP}* { iface_factorize(); }
+^{SP}*(seq|sequentialize){SP}* { iface_sequentialize(); }
+^{SP}*(h(e(l(p{SP}+)?)?)?)?(licen(c|s)e|warranty) { iface_warranty(); }
+^{SP}*help{SP}* { iface_help(); }
+^{SP}*help{SP}+/[^ ] { BEGIN(HELP); }
+^{SP}*igno(re?)({SP}+net?)? { iface_ignore(); }
+^{SP}*intersect({SP}+net?)? { iface_intersect();}
+^{SP}*inv(e(rt?)?)?({SP}+net?)? { iface_invert(); }
+^{SP}*label{SP}+net{SP}* { iface_label_net(); }
+^{SP}*letter{SP}+machine{SP}* { iface_letter_machine(); }
+^{SP}*(load{SP}+defined{SP}+|loadd{SP}+){SP}* { BEGIN(LOAD_DEFINED); }
+^{SP}*(load{SP}+stack{SP}+?|load{SP}+){SP}* { BEGIN(LOAD_STACK); }
+^{SP}*lower\-side({SP}+net?)? { iface_lower_side();}
+^{SP}*min(imize)?({SP}+net?)? { iface_minimize(); }
+^{SP}*na(me?)?({SP}+net?)?{SP}+ { BEGIN(NAME_NET); }
+^{SP}*neg(a(te?)?)?({SP}+net?)? { iface_negate(); }
+^{SP}*on(e(\-(p(l(us?)?)?)?)?)?({SP}+net?)? { iface_one_plus();}
+^{SP}*pop?|pop{SP}+st(a(ck?)?)? { iface_pop();}
+^{SP}*(pr(i(nt?)?)?{SP}+)?cma(t(r(ix?)?)?)? { iface_print_cmatrix(); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?def(i(n(ed?)?)?)? { iface_print_defined(); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?dot{SP}* { iface_print_dot(NULL); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?dot{SP}*>{SP}*/[^ >] { BEGIN(PRINT_DOT_FILE); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?dot{SP}+/[^ >] { BEGIN(PRINT_DOT_NAME); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?low(e(r\-(w(o(r(ds?)?)?)?)?)?)? { iface_lower_words(-1); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?low(e(r\-(w(o(r(ds?)?)?)?)?)?)?{SP}+[0-9]+ { iface_lower_words(iface_extract_number(interfacetext)); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?na(me?)? { iface_print_name(); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?net{SP}* { iface_print_net(NULL,NULL); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?net{SP}*>{SP}*/[^ >] { BEGIN(PRINT_NET_FILE); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?net{SP}+/[^ >\t] { BEGIN(PRINT_NET_NAME); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?net{SP}+/[^ >\t][ \t]*>[ \t]*[^ >] { BEGIN(PRINT_NET_NAME_FILE); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-lower { iface_random_lower(-1); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-lower{SP}+[0-9]+ { iface_random_lower(iface_extract_number(interfacetext)); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-upper { iface_random_upper(-1); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-upper{SP}+[0-9]+ { iface_random_upper(iface_extract_number(interfacetext)); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-words { iface_random_words(-1); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-words{SP}+[0-9]+ { iface_random_words(iface_extract_number(interfacetext)); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?sig(ma?)? { iface_print_sigma(); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?size? { iface_print_stats(); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?stack-size? { printf("STACK SIZE: %d\n", stack_size()); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?upp(e(r(\-(w(o(r(ds?)?)?)?)?)?)?)? { iface_upper_words(-1); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?upp(e(r(\-(w(o(r(ds?)?)?)?)?)?)?)?{SP}+[0-9]+ { iface_upper_words(iface_extract_number(interfacetext)); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?pairs { iface_pairs(-1);}
+^{SP}*(pr(i(nt?)?)?{SP}+)?random\-pairs { iface_random_pairs(-1);}
+^{SP}*(pr(i(nt?)?)?{SP}+)?words { iface_words(-1);}
+^{SP}*(pr(i(nt?)?)?{SP}+)?words{SP}+[0-9]+ { iface_words(iface_extract_number(interfacetext));}
+^{SP}*(pr(i(nt?)?)?{SP}+)?words{SP}*>{SP}* { BEGIN(WORDS_FILE); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?pairs{SP}*>{SP}* { BEGIN(PAIRS_FILE); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?low(e(r\-(w(o(r(ds?)?)?)?)?)?)?{SP}*>{SP}* { BEGIN(LOWER_WORDS_FILE); }
+^{SP}*(pr(i(nt?)?)?{SP}+)?upp(e(r\-(w(o(r(ds?)?)?)?)?)?)?{SP}*>{SP}* { BEGIN(UPPER_WORDS_FILE); }
+^{SP}*((pr(i(nt?)?)?{SP}+)?shortest-string|pss) { iface_print_shortest_string();}
+^{SP}*((pr(i(nt?)?)?{SP}+)?shortest-string-size|psz) { iface_print_shortest_string_size();}
+^{SP}*pru(ne?)?({SP}+net?)? { iface_prune(); }
+^{SP}*pu(sh?)?({SP}+def(i(n(ed?)?)?)?)?{SP}* { BEGIN(PUSH); }
+^{SP}*(quit|au{SP}+revoir|bye|exit|hyv[\303][\244]sti) { iface_quit(); }
+^{SP}*(re(ad?)?{SP}+)?att{SP}*<?{SP}*|ratt{SP}*<?{SP}* { BEGIN(ATT); }
+^{SP}*(re(ad?)?{SP}+)?cmatrix{SP}* { printf("Missing filename.\n"); }
+^{SP}*(re(ad?)?{SP}+)?cmatrix{SP}*<?{SP}*/[^ \t] { BEGIN(RCMATRIX); }
+^{SP}*(re(ad?)?{SP}+)?prol(og?)?{SP}*<?{SP}*|rpl{SP}*<?{SP}* { BEGIN(RPL); }
+^{SP}*(re(ad?)?{SP}+)?lexc{SP}*<?{SP}* { BEGIN(RLEXC); }
+^{SP}*(re(ad?)?{SP}+)?re(g(ex?)?)?{SP}+ { pmode = RE; BEGIN(REGEX); }
+^{SP}*(re(ad?)?{SP}+)?spaced\-text{SP}*<?{SP}* { BEGIN(READ_SPACED_TEXT); }
+^{SP}*(re(ad?)?{SP}+)?text{SP}*<?{SP}* { BEGIN(READ_TEXT); }
+^{SP}*rev(e(r(se?)?)?)?({SP}+net?)? { iface_reverse(); }
+^{SP}*rot(a(te?)?)?({SP}+st(a(ck?)?)?)? { iface_rotate();}
+^{SP}*(save{SP}+defined{SP}+|saved{SP}+)>?{SP}* { BEGIN(SAVE_DEFINED); }
+^{SP}*(save{SP}+stack{SP}+?|ss{SP}+)>?{SP}* { BEGIN(SAVE_STACK); }
+^{SP}*set[ ]+/[^ ]+[ ]+[^ ]+ { BEGIN(SET_VAR); }
+^{SP}*show{SP}+var(i(a(b(l(es?)?)?)?)?)? { iface_show_variables();}
+^{SP}*show({SP}+(v(a(r(i(a(b(l(e)?)?)?)?)?)?)?)?)?{SP}+/[^ ] { BEGIN(SHOW_VAR); }
+^{SP}*shu(f(f(le?)?)?)?({SP}+net?)? { iface_shuffle(); }
+^{SP}*sigma{SP}+net{SP}* { iface_sigma_net(); }
+^{SP}*so(rt?)?({SP}+in(put)?) { iface_sort_input(); }
+^{SP}*so(rt?)?({SP}+out(put)?) { iface_sort_output(); }
+^{SP}*so(rt?)?({SP}+net?)? { iface_sort(); }
+^{SP}*sou(r(ce?)?)?{SP}+ {BEGIN(SOURCE);}
+^{SP}*sub(s(t(i(t(u(te?)?)?)?)?)?)?{SP}+def(i(n(ed?)?)?)?{SP}+ {BEGIN(SUBSTITUTE_DEFINED);}
+^{SP}*sub(s(t(i(t(u(te?)?)?)?)?)?)?{SP}+sym(b(ol?)?)?{SP}+ {BEGIN(SUBSTITUTE_SYMBOL);}
+^{SP}*sy(s(t(em?)?)?)?{SP}+ {BEGIN(SYSTEM);}
+^{SP}*(test{SP}+unambiguous|tunam) { iface_test_unambiguous(); }
+^{SP}*(test{SP}+equivalent|equ) { iface_test_equivalent(); }
+^{SP}*(test{SP}+functional|tfu) { iface_test_functional(); }
+^{SP}*(test{SP}+identity|tid) { iface_test_identity(); }
+^{SP}*(test{SP}+non-null|tnn) { iface_test_nonnull(); }
+^{SP}*(test{SP}+null|tnu) { iface_test_null(); }
+^{SP}*(test{SP}+lower-universal|tlu) { iface_test_lower_universal(); }
+^{SP}*(test{SP}+sequential|tseq) { iface_test_sequential(); }
+^{SP}*(test{SP}+upper-univesal|tuu) { iface_test_upper_universal(); }
+^{SP}*tu(rn?)?({SP}+st(a(ck?)?)?)? { iface_turn(); }
+^{SP}*(twosided{SP}+flag-diacritics|tfd) { iface_twosided_flags(); }
+^{SP}*und(e(f(i(ne?)?)?)?)?{SP}+ {BEGIN(UNDEFINE);}
+^{SP}*uni(on?)?({SP}+net?)? {iface_union();}
+^{SP}*upper\-side({SP}+n(et?)?)? { iface_upper_side(); }
+^{SP}*view({SP}+net?)? { iface_view(); }
+^{SP}*(((wr(i(te?)?)?{SP}+)?prolog)|(wpl)){SP}* { iface_write_prolog(NULL); }
+^{SP}*(((wr(i(te?)?)?{SP}+)?prolog)|(wpl)){SP}+>?{SP}*/[^ >] { BEGIN(WRITE_PROLOG_FILE); }
+^{SP}*(((wr(i(te?)?)?{SP}+)?att)|(watt)){SP}* { iface_write_att(NULL); }
+^{SP}*(((wr(i(te?)?)?{SP}+)?att)|(watt)){SP}+>?{SP}*/[^ >] { BEGIN(WRITE_ATT_FILE); }
+^{SP}*ze(r(o(\-(p(l(us?)?)?)?)?)?)?({SP}+net?)? { iface_zero_plus(); }
+
+<APPLY_FILE_IN>{NONL}+/[>] {
+ tempstr = trim(strdup(interfacetext));
+ BEGIN(APPLY_FILE_EATUP);
+}
+
+<APPLY_FILE_IN>[^>]+ {
+ iface_apply_file(trim(interfacetext),NULL, applydir);
+ BEGIN(INITIAL);
+}
+<APPLY_FILE_EATUP>[>][ ]* {
+ BEGIN(APPLY_FILE_OUT);
+}
+<APPLY_FILE_OUT>{NONL}+ {
+ iface_apply_file(tempstr, trim(interfacetext), applydir);
+ xxfree(tempstr);
+ tempstr = NULL;
+ BEGIN(INITIAL);
+}
+
+<APPLY_DOWN>{NONL}+ {
+ iface_apply_down(interfacetext);
+ BEGIN(INITIAL);
+}
+<APPLY_DOWN>{NONL}+ {
+ iface_apply_down(interfacetext);
+ BEGIN(INITIAL);
+}
+<APPLY_MED>{NONL}+ {
+ iface_apply_med(interfacetext);
+ BEGIN(INITIAL);
+}
+<APPLY_UP>{NONL}+ {
+ iface_apply_up(interfacetext);
+ BEGIN(INITIAL);
+}
+<NAME_NET>{NONL}+ {
+ iface_name_net(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<HELP>{NONL}+ {
+ iface_help_search(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<APROPOS>{NONL}+ {
+ iface_apropos(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<PUSH>{NONL}+ {
+ if (find_defined(g_defines, interfacetext) == NULL)
+ printf("'%s' is not a defined symbol.\n",interfacetext);
+ else
+ stack_add(fsm_copy(find_defined(g_defines, interfacetext)));
+ BEGIN(INITIAL);
+}
+
+<APPLY_P>{NONL}+ {
+ if (strcmp(interfacetext,"END;") == 0) {
+ promptmode = PROMPT_MAIN;
+ BEGIN(INITIAL);
+ } else {
+ if (apply_direction == AP_D)
+ iface_apply_down(interfacetext);
+
+ if (apply_direction == AP_M)
+ iface_apply_med(interfacetext);
+
+ if (apply_direction == AP_U)
+ iface_apply_up(interfacetext);
+ }
+}
+
+
+<REGEX>(#|!) { yymore(); BEGIN(RCOMMENT); }
+
+(#|!).* { }
+
+<REGEX>(\.#) { yymore();}
+
+<REGEX>({INREGEX}|%{ANY}) {
+ yymore();
+}
+
+<RCOMMENT>[\012] { yymore(); BEGIN(REGEX); }
+<RCOMMENT>{ANY} { yymore(); }
+
+<REGEX>(;) {
+ if (my_yyparse(interfacetext, interfacelineno, g_defines, g_defines_f) == 0) {
+ /* regex xxx line */
+ if (pmode == RE) {
+ stack_add(fsm_topsort(fsm_minimize(current_parse)));
+ /* define XXX xxx line */
+ } else if (pmode == DE) {
+ tempnet = fsm_topsort(fsm_minimize(current_parse));
+ olddef = add_defined(g_defines, tempnet,tempstr);
+ if (olddef) {
+ printf("redefined %s: ",tempstr);
+ } else {
+ printf("defined %s: ",tempstr);
+ }
+ print_stats(tempnet);
+ xxfree(tempstr);
+ tempstr = NULL;
+ }
+ }
+ BEGIN(INITIAL);
+}
+<REGEX>[{] {
+ BEGIN(REGEXB);
+ yymore();
+}
+<REGEXB>[^}] {
+ yymore();
+}
+<REGEXB>[}] {
+ BEGIN(REGEX);
+ yymore();
+}
+<REGEX>(["]) {
+ BEGIN(REGEXQ);
+ yymore();
+}
+<REGEXQ>([^"]*) {
+ yymore();
+}
+<REGEXQ>([\042]) {
+ BEGIN(REGEX);
+ yymore();
+}
+
+<DEFI>[^ \t(]+/[\050] {
+ func_name = xxmalloc(sizeof(char)*(strlen(interfacetext)+2));
+ func_name = strcpy(func_name, interfacetext);
+ strcat(func_name, "(");
+ func_arg_no = 1;
+ BEGIN(FUNC_1);
+}
+
+ /* Eat up parenthesis */
+<FUNC_1>\( { BEGIN(FUNC_2); }
+
+<FUNC_2>[ ] { }
+<FUNC_2>[^ ,)]+/[ ]*, {
+ add_func_arg(interfacetext);
+}
+<FUNC_2>[^,) ]+/[ ]*\) {
+ add_func_arg(interfacetext);
+ BEGIN(FUNC_3);
+}
+
+<FUNC_3>\)[ \t]*+ { funcdef[0] = '\0'; BEGIN(FUNC_4); }
+<FUNC_4>{NONRESERVED}+ { tempstr = rep_func_arg(interfacetext); strcat(funcdef, tempstr); xxfree(tempstr); tempstr = NULL;}
+<FUNC_4>{BRACED} { strcat(funcdef, interfacetext); }
+<FUNC_4>%{ANY} { strcat(funcdef, interfacetext); }
+<FUNC_4>{QUOTED} { strcat(funcdef, interfacetext); }
+<FUNC_4>; { strcat(funcdef, interfacetext); add_defined_function(g_defines_f, func_name, funcdef, (func_arg_no-1));clear_func_args(); xxfree(func_name); BEGIN(INITIAL); }
+<FUNC_4>.|\.#\. { strcat(funcdef, interfacetext); }
+
+
+<DEFI>([\001-\010]|[\013-\037]|[\041-\047]|[\051-\072]|[\074-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277])+/[ \t]+ {
+ tempstr = xxstrdup(interfacetext);
+ BEGIN(REGEX);
+}
+
+<DEFI>(([\001-\010]|[\013-\037]|[\041-\047]|[\051-\072]|[\074-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277])+)[ \t]+?;? {
+
+ tempnet = NULL;
+ /* Define the top network on stack */
+ if (iface_stack_check(1)) {
+ tempnet = stack_pop();
+ olddef = add_defined(g_defines, tempnet,remove_trailing(interfacetext,';'));
+ if (olddef) {
+ printf("redefined %s: ",interfacetext);
+ } else {
+ printf("defined %s: ",interfacetext);
+ }
+ print_stats(tempnet);
+ }
+ BEGIN(INITIAL);
+}
+
+<UNDEFINE>{NONL}+ {
+ remove_defined(g_defines, remove_trailing(interfacetext,';'));
+ BEGIN(INITIAL);
+}
+
+<PRINT_NET_FILE>{NONL}+ {
+ iface_print_net(NULL,trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<PRINT_NET_NAME>{NONL}+ {
+ iface_print_net(trim(interfacetext), NULL);
+ BEGIN(INITIAL);
+}
+<PRINT_NET_NAME_FILE>[^ \t]+ {
+ tempstr = strdup(trim(interfacetext));
+ BEGIN(PRINT_NET_NAME_FILE2);
+}
+<PRINT_NET_NAME_FILE2>[> \t] { }
+<PRINT_NET_NAME_FILE2>[^ \t>]+ {
+ iface_print_net(tempstr, trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<WRITE_ATT_FILE>{NONL}+ {
+ iface_write_att(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<WRITE_PROLOG_FILE>{NONL}+ {
+ iface_write_prolog(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<PRINT_DOT_FILE>{NONL}+ {
+ iface_print_dot(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<SOURCE>{NONL}+ {
+ if ((yyin = fopen(trim(interfacetext), "r" )) != NULL) {
+ printf("Opening file '%s'.\n", trim(interfacetext));
+ input_is_file = 1;
+ yypush_buffer_state(yy_create_buffer(yyin, YY_BUF_SIZE*2));
+ } else {
+ printf("Error opening file '%s'\n",trim(interfacetext));
+ }
+ BEGIN(INITIAL);
+}
+
+<SHOW_VAR>{NONL}+ {
+ iface_show_variable(interfacetext);
+ BEGIN(INITIAL);
+}
+<SET_VAR>[^ ]+ {
+ tempstr = xxstrdup(interfacetext);
+ BEGIN(SET_VALUE);
+}
+<SET_VALUE>[^ \n]+ {
+ iface_set_variable(tempstr,interfacetext);
+ xxfree(tempstr);
+ tempstr = NULL;
+ BEGIN(INITIAL);
+}
+<*>[ ] { }
+
+<READ_SPACED_TEXT>{NONL}+ {
+ iface_read_spaced_text(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<READ_TEXT>{NONL}+ {
+ iface_read_text(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<WORDS_FILE>{NONL}+ {
+ iface_words_file(trim(interfacetext),0);
+ BEGIN(INITIAL);
+}
+
+<PAIRS_FILE>{NONL}+ {
+ iface_pairs_file(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<UPPER_WORDS_FILE>{NONL}+ {
+ iface_words_file(trim(interfacetext),1);
+ BEGIN(INITIAL);
+}
+
+<LOWER_WORDS_FILE>{NONL}+ {
+ iface_words_file(trim(interfacetext),2);
+ BEGIN(INITIAL);
+}
+
+
+<SAVE_DEFINED>{NONL}+ {
+ iface_save_defined(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<SAVE_STACK>{NONL}+ {
+ iface_save_stack(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<LOAD_STACK>{NONL}+ {
+ iface_load_stack(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<LOAD_DEFINED>{NONL}+ {
+ iface_load_defined(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+<SUBSTITUTE_SYMBOL>[^ \t]+ { tempstr = xxstrdup(interfacetext); BEGIN(SUBSTITUTE_SYMBOL_2);}
+<SUBSTITUTE_SYMBOL_2>for{SP}+ {BEGIN(SUBSTITUTE_SYMBOL_3);}
+<SUBSTITUTE_SYMBOL_3>{NONL}+ {
+ iface_substitute_symbol(interfacetext, tempstr);
+ BEGIN(INITIAL);
+}
+<SUBSTITUTE_DEFINED>[^ \t]+ { tempstr = xxstrdup(interfacetext); BEGIN(SUBSTITUTE_DEFINED_2);}
+<SUBSTITUTE_DEFINED_2>for{SP}+ {BEGIN(SUBSTITUTE_DEFINED_3);}
+<SUBSTITUTE_DEFINED_3>{NONL}+ {
+ iface_substitute_defined(interfacetext, tempstr);
+ BEGIN(INITIAL);
+}
+
+<ELIMINATE_FLAG>{NONL}+ {
+ iface_eliminate_flag(interfacetext);
+ BEGIN(INITIAL);
+}
+
+<SYSTEM>{NONL}+ {
+ int ret;
+ ret = system(interfacetext);
+ BEGIN(INITIAL);
+}
+
+<ECHO>{NONL}+ {
+ printf("%s\n",interfacetext);
+ BEGIN(INITIAL);
+}
+
+<ASSERT_STACK>{NONL} {
+ int level = strtoul(interfacetext, 0, 10);
+ if (level != stack_size()) {
+ fprintf(stderr, "Stack size %d not %d\n", stack_size(), level);
+ exit(1);
+ }
+ BEGIN(INITIAL);
+}
+
+<ATT>{NONL}+ {
+ iface_read_att(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<RPL>{NONL}+ {
+ iface_read_prolog(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<RLEXC>{NONL}+ {
+ if ((lexcfilein = file_to_mem(trim(interfacetext))) != NULL) {
+ stack_add(fsm_lexc_parse_string(lexcfilein, 1));
+ xxfree(lexcfilein);
+ } else {
+ printf("Error opening file '%s'.\n", interfacetext);
+ }
+ BEGIN(INITIAL);
+}
+
+<EXCMATRIX>{NONL}+ {
+ iface_print_cmatrix_att(trim(interfacetext));
+ BEGIN(INITIAL);
+}
+
+<RCMATRIX>{NONL}+ {
+ if (iface_stack_check(1)) {
+ if ((cmatrixfilein = file_to_mem(trim(interfacetext))) != NULL) {
+ printf("Reading confusion matrix from file '%s'\n",interfacetext);
+ my_cmatrixparse(stack_find_top()->fsm, cmatrixfilein);
+ xxfree(cmatrixfilein);
+ } else {
+ perror("File error");
+ }
+ }
+ BEGIN(INITIAL);
+}
+
+([\040]|[\011]|[\015]) { }
+
+<*>[\012] { }
+<*>[\015] { }
+
+<INITIAL>[^#!] {
+ if (!input_is_file)
+ printf("Unknown command. Ignoring until end of line.\n");
+ else
+ printf("***Unknown command '%s' on line %i. Aborting.\n",interfacetext,interfacelineno);
+ return 1;
+}
+
+<IGNORELINE>.? { BEGIN(INITIAL); }
+
+<<EOF>> {
+ yypop_buffer_state();
+ if (!YY_CURRENT_BUFFER) {
+ yyterminate();
+ }
+}
diff --git a/io.c b/io.c
new file mode 100644
index 0000000..cc8d755
--- /dev/null
+++ b/io.c
@@ -0,0 +1,1013 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <stdarg.h>
+#include "foma.h"
+#include "zlib.h"
+
+#define TYPE_TRANSITION 1
+#define TYPE_SYMBOL 2
+#define TYPE_FINAL 3
+#define TYPE_PROPERTY 4
+#define TYPE_END 5
+#define TYPE_ERROR 6
+
+#define READ_BUF_SIZE 4096
+
+struct binaryline {
+ int type;
+ int state;
+ int in;
+ int target;
+ int out;
+ int symbol;
+ char *name;
+ char *value;
+};
+
+extern char *g_att_epsilon;
+
+extern struct defined_networks *g_defines;
+extern struct defined_functions *g_defines_f;
+
+struct io_buf_handle {
+ char *io_buf;
+ char *io_buf_ptr;
+};
+
+struct io_buf_handle *io_init();
+void io_free(struct io_buf_handle *iobh);
+static int io_gets(struct io_buf_handle *iobh, char *target);
+static size_t io_get_gz_file_size(char *filename);
+static size_t io_get_file_size(char *filename);
+static size_t io_get_regular_file_size(char *filename);
+size_t io_gz_file_to_mem (struct io_buf_handle *iobh, char *filename);
+int foma_net_print(struct fsm *net, gzFile outfile);
+struct fsm *io_net_read(struct io_buf_handle *iobh, char **net_name);
+static INLINE int explode_line (char *buf, int *values);
+
+
+void escape_print(FILE *stream, char* string) {
+ int i;
+ if (strchr(string, '"') != NULL) {
+ for (i = 0; *(string+i) != '\0'; i++) {
+ if (*(string+i) == '"') {
+ fprintf(stream, "\\\"");
+ } else {
+ fputc(*(string+i), stream);
+ }
+ }
+ } else {
+ fprintf(stream, "%s", string);
+ }
+}
+
+int foma_write_prolog (struct fsm *net, char *filename) {
+ struct fsm_state *stateptr;
+ int i, *finals, *used_symbols, maxsigma;
+ FILE *out;
+ char *outstring, *instring, identifier[100];
+
+ if (filename == NULL) {
+ out = stdout;
+ } else {
+ if ((out = fopen(filename, "w")) == NULL) {
+ printf("Error writing to file '%s'. Using stdout.\n", filename);
+ out = stdout;
+ }
+ printf("Writing prolog to file '%s'.\n", filename);
+ }
+ fsm_count(net);
+ maxsigma = sigma_max(net->sigma);
+ used_symbols = xxcalloc(maxsigma+1,sizeof(int));
+ finals = xxmalloc(sizeof(int)*(net->statecount));
+ stateptr = net->states;
+ identifier[0] = '\0';
+
+ strcpy(identifier, net->name);
+
+ /* Print identifier */
+ fprintf(out, "%s%s%s", "network(",identifier,").\n");
+
+ for (i=0; (stateptr+i)->state_no != -1; i++) {
+ if ((stateptr+i)->final_state == 1) {
+ *(finals+((stateptr+i)->state_no)) = 1;
+ } else {
+ *(finals+((stateptr+i)->state_no)) = 0;
+ }
+ if ((stateptr+i)->in != -1) {
+ *(used_symbols+((stateptr+i)->in)) = 1;
+ }
+ if ((stateptr+i)->out != -1) {
+ *(used_symbols+((stateptr+i)->out)) = 1;
+ }
+
+ }
+
+ for (i = 3; i <= maxsigma; i++) {
+ if (*(used_symbols+i) == 0) {
+ instring = sigma_string(i, net->sigma);
+ if (strcmp(instring,"0") == 0) {
+ instring = "%0";
+ }
+ fprintf(out, "symbol(%s, \"", identifier);
+ escape_print(out, instring);
+ fprintf(out, "\").\n");
+
+ }
+ }
+
+ for (; stateptr->state_no != -1; stateptr++) {
+ if (stateptr->target == -1)
+ continue;
+ fprintf(out, "arc(%s, %i, %i, ", identifier, stateptr->state_no, stateptr->target);
+ if (stateptr->in == 0) instring = "0";
+ else if (stateptr->in == 1) instring = "?";
+ else if (stateptr->in == 2) instring = "?";
+ else instring = sigma_string(stateptr->in, net->sigma);
+ if (stateptr->out == 0) outstring = "0";
+ else if (stateptr->out == 1) outstring = "?";
+ else if (stateptr->out == 2) outstring = "?";
+ else outstring = sigma_string(stateptr->out, net->sigma);
+
+ if (strcmp(instring,"0") == 0 && stateptr->in != 0) instring = "%0";
+ if (strcmp(outstring,"0") == 0 && stateptr->out != 0) outstring = "%0";
+ if (strcmp(instring,"?") == 0 && stateptr->in > 2) instring = "%?";
+ if (strcmp(outstring,"?") == 0 && stateptr->in > 2) outstring = "%?";
+ /* Escape quotes */
+
+ if (net->arity == 2 && stateptr->in == IDENTITY && stateptr->out == IDENTITY) {
+ fprintf(out, "\"?\").\n");
+ }
+ else if (net->arity == 2 && stateptr->in == stateptr->out && stateptr->in != UNKNOWN) {
+ fprintf(out, "\"");
+ escape_print(out, instring);
+ fprintf(out, "\").\n");
+ }
+ else if (net->arity == 2) {
+ fprintf(out, "\"");
+ escape_print(out, instring);
+ fprintf(out, "\":\"");
+ escape_print(out, outstring);
+ fprintf(out, "\").\n");
+ }
+ else if (net->arity == 1) {
+ fprintf(out, "\"");
+ escape_print(out, instring);
+ fprintf(out, "\").\n");
+ }
+ }
+
+ for (i = 0; i < net->statecount; i++) {
+ if (*(finals+i)) {
+ fprintf(out, "final(%s, %i).\n", identifier, i);
+ }
+ }
+ if (filename != NULL) {
+ fclose(out);
+ }
+ xxfree(finals);
+ xxfree(used_symbols);
+ return 1;
+}
+
+struct fsm *read_att(char *filename) {
+
+ struct fsm_construct_handle *h;
+ struct fsm *net;
+ int i;
+ char inword[1024], delimiters[] = "\t", *tokens[6];
+ FILE *INFILE;
+
+ INFILE = fopen(filename, "r");
+ if (INFILE == NULL) {
+ return(NULL);
+ }
+
+ h = fsm_construct_init(filename);
+ while (fgets(inword, 1024, INFILE) != NULL) {
+ if (inword[strlen(inword)-1] == '\n') {
+ inword[strlen(inword)-1] = '\0';
+ }
+ tokens[0] = strtok(inword, delimiters);
+ i = 0;
+ if (tokens[0] != NULL) {
+ i = 1;
+ for ( ; ; ) {
+ tokens[i] = strtok(NULL, delimiters);
+ if (tokens[i] == NULL) {
+ break;
+ }
+ i++;
+ if (i == 6)
+ break;
+ }
+ }
+ if (i == 0) { continue; }
+ if (i >= 4) {
+ if (strcmp(tokens[2],g_att_epsilon) == 0)
+ tokens[2] = "@_EPSILON_SYMBOL_@";
+ if (strcmp(tokens[3],g_att_epsilon) == 0)
+ tokens[3] = "@_EPSILON_SYMBOL_@";
+
+ fsm_construct_add_arc(h, atoi(tokens[0]), atoi(tokens[1]), tokens[2], tokens[3]);
+ }
+ else if (i <= 3 && i > 0) {
+ fsm_construct_set_final(h,atoi(tokens[0]));
+ }
+ }
+ fsm_construct_set_initial(h,0);
+ fclose(INFILE);
+ net = fsm_construct_done(h);
+ fsm_count(net);
+ net = fsm_topsort(net);
+ return(net);
+}
+
+struct fsm *fsm_read_prolog (char *filename) {
+ char buf [1024], temp [1024], in [128], out[128], *temp_ptr, *temp_ptr2;
+ int arity, source, target, has_net;
+ struct fsm *outnet;
+ struct fsm_construct_handle *outh = NULL;
+ FILE *prolog_file;
+
+ has_net = 0;
+ prolog_file = fopen(filename, "r");
+ if (prolog_file == NULL) {
+ return NULL;
+ }
+
+ while (fgets(buf, 1023, prolog_file) != NULL) {
+ if (strstr(buf, "network(") == buf) {
+ /* Extract network name */
+ if (has_net == 1) {
+ perror("WARNING: prolog file contains multiple nets. Only returning the first one.\n");
+ break;
+ } else {
+ has_net = 1;
+ }
+ temp_ptr = strstr(buf, "network(")+8;
+ temp_ptr2 = strstr(buf, ").");
+ strncpy(temp, temp_ptr, (temp_ptr2 - temp_ptr));
+ temp[(temp_ptr2-temp_ptr)] = '\0';
+
+ /* Start network */
+ outh = fsm_construct_init(temp);
+ }
+ if (strstr(buf, "final(") == buf) {
+ temp_ptr = strstr(buf, " ");
+ temp_ptr++;
+ temp_ptr2 = strstr(temp_ptr, ").");
+ strncpy(temp, temp_ptr, (temp_ptr2 - temp_ptr));
+ temp[(temp_ptr2-temp_ptr)] = '\0';
+
+ fsm_construct_set_final(outh, atoi(temp));
+ }
+ if (strstr(buf, "symbol(") == buf) {
+ temp_ptr = strstr(buf, ", \"")+3;
+ temp_ptr2 = strstr(temp_ptr, "\").");
+ strncpy(temp, temp_ptr, (temp_ptr2 - temp_ptr));
+ temp[(temp_ptr2-temp_ptr)] = '\0';
+ if (strcmp(temp, "%0") == 0)
+ strcpy(temp, "0");
+ //printf("special: %s\n",temp);
+
+ if (fsm_construct_check_symbol(outh, temp) == -1) {
+ fsm_construct_add_symbol(outh, temp);
+ }
+ continue;
+ }
+ if (strstr(buf, "arc(") == buf) {
+ in[0] = '\0';
+ out[0] = '\0';
+
+ if (strstr(buf, "\":\"") == NULL || strstr(buf, ", \":\").") != NULL) {
+ arity = 1;
+ } else {
+ arity = 2;
+ }
+
+ /* Get source */
+ temp_ptr = strstr(buf, " ");
+ temp_ptr++;
+ temp_ptr2 = strstr(temp_ptr, ",");
+ strncpy(temp, temp_ptr, (temp_ptr2 - temp_ptr));
+ temp[(temp_ptr2-temp_ptr)] = '\0';
+ source = atoi(temp);
+
+ /* Get target */
+ temp_ptr = strstr(temp_ptr2, " ");
+ temp_ptr++;
+ temp_ptr2 = strstr(temp_ptr, ",");
+ strncpy(temp, temp_ptr, (temp_ptr2 - temp_ptr));
+ temp[(temp_ptr2-temp_ptr)] = '\0';
+ target = atoi(temp);
+
+ temp_ptr = strstr(temp_ptr2, "\"");
+ temp_ptr++;
+ if (arity == 2) {
+ temp_ptr2 = strstr(temp_ptr, "\":");
+ } else {
+ temp_ptr2 = strstr(temp_ptr, "\").");
+ }
+ strncpy(in, temp_ptr, (temp_ptr2 - temp_ptr));
+ in[(temp_ptr2 - temp_ptr)] = '\0';
+
+ if (arity == 2) {
+ temp_ptr = strstr(temp_ptr2, ":\"");
+ temp_ptr += 2;
+ temp_ptr2 = strstr(temp_ptr, "\").");
+ strncpy(out, temp_ptr, (temp_ptr2 - temp_ptr));
+ out[(temp_ptr2 - temp_ptr)] = '\0';
+ }
+ if (arity == 1 && (strcmp(in, "?") == 0)) {
+ strcpy(in,"@_IDENTITY_SYMBOL_@");
+ }
+ if (arity == 2 && (strcmp(in, "?") == 0)) {
+ strcpy(in,"@_UNKNOWN_SYMBOL_@");
+ }
+ if (arity == 2 && (strcmp(out, "?") == 0)) {
+ strcpy(out,"@_UNKNOWN_SYMBOL_@");
+ }
+ if (strcmp(in, "0") == 0) {
+ strcpy(in,"@_EPSILON_SYMBOL_@");
+ }
+ if (strcmp(out, "0") == 0) {
+ strcpy(out,"@_EPSILON_SYMBOL_@");
+ }
+ if (strcmp(in, "%0") == 0) {
+ strcpy(in,"0");
+ }
+ if (strcmp(out, "%0") == 0) {
+ strcpy(out,"0");
+ }
+ if (strcmp(in, "%?") == 0) {
+ strcpy(in,"?");
+ }
+ if (strcmp(out, "%?") == 0) {
+ strcpy(out,"?");
+ }
+
+ if (arity == 1) {
+ fsm_construct_add_arc(outh, source, target, in, in);
+ } else {
+ fsm_construct_add_arc(outh, source, target, in, out);
+ }
+ }
+ }
+ fclose(prolog_file);
+ if (has_net == 1) {
+ fsm_construct_set_initial(outh, 0);
+ outnet = fsm_construct_done(outh);
+ fsm_topsort(outnet);
+ return(outnet);
+ } else {
+ return(NULL);
+ }
+}
+
+struct io_buf_handle *io_init() {
+ struct io_buf_handle *iobh;
+ iobh = xxmalloc(sizeof(struct io_buf_handle));
+ (iobh->io_buf) = NULL;
+ (iobh->io_buf_ptr) = NULL;
+ return(iobh);
+}
+
+void io_free(struct io_buf_handle *iobh) {
+ if (iobh->io_buf != NULL) {
+ xxfree(iobh->io_buf);
+ (iobh->io_buf) = NULL;
+ }
+ xxfree(iobh);
+}
+
+char *spacedtext_get_next_line(char **text) {
+ char *t, *ret;
+ ret = *text;
+ if (**text == '\0')
+ return NULL;
+ for (t = *text; *t != '\0' && *t != '\n'; t++) {
+ }
+ if (*t == '\0')
+ *text = t;
+ else
+ *text = t+1;
+ *t = '\0';
+ return(ret);
+}
+
+char *spacedtext_get_next_token(char **text) {
+ char *t, *ret;
+ if (**text == '\0' || **text == '\n')
+ return NULL;
+ for ( ; **text == ' ' ; (*text)++) {
+ }
+ ret = *text;
+ for (t = *text; *t != '\0' && *t != '\n' && *t != ' '; t++) {
+ }
+ if (*t == '\0' || *t == '\n')
+ *text = t;
+ else
+ *text = t+1;
+ *t = '\0';
+ return(ret);
+}
+
+struct fsm *fsm_read_spaced_text_file(char *filename) {
+ struct fsm_trie_handle *th;
+ char *text, *textorig, *insym, *outsym, *t1, *t2, *l1, *l2;
+
+ text = textorig = file_to_mem(filename);
+
+ if (text == NULL)
+ return NULL;
+ th = fsm_trie_init();
+ for (;;) {
+ for ( ; *text != '\0' && *text == '\n'; text++) { }
+ t1 = spacedtext_get_next_line(&text);
+ if (t1 == NULL)
+ break;
+ if (strlen(t1) == 0)
+ continue;
+ t2 = spacedtext_get_next_line(&text);
+ if (t2 == NULL || strlen(t2) == 0) {
+ for (l1 = t1; (insym = spacedtext_get_next_token(&l1)) != NULL; ) {
+ if (strcmp(insym, "0") == 0)
+ fsm_trie_symbol(th, "@_EPSILON_SYMBOL_@", "@_EPSILON_SYMBOL_@");
+ else if (strcmp(insym, "%0") == 0)
+ fsm_trie_symbol(th, "0", "0");
+ else
+ fsm_trie_symbol(th, insym, insym);
+ }
+ fsm_trie_end_word(th);
+ } else {
+ for (l1 = t1, l2 = t2; ; ) {
+ insym = spacedtext_get_next_token(&l1);
+ outsym = spacedtext_get_next_token(&l2);
+ if (insym == NULL && outsym == NULL)
+ break;
+ if (insym == NULL || strcmp(insym, "0") == 0)
+ insym = "@_EPSILON_SYMBOL_@";
+ if (strcmp(insym, "%0") == 0)
+ insym = "0";
+ if (outsym == NULL || strcmp(outsym, "0") == 0)
+ outsym = "@_EPSILON_SYMBOL_@";
+ if (strcmp(outsym, "%0") == 0)
+ outsym = "0";
+ fsm_trie_symbol(th, insym, outsym);
+ }
+ fsm_trie_end_word(th);
+ }
+ }
+ xxfree(textorig);
+ return(fsm_trie_done(th));
+}
+
+struct fsm *fsm_read_text_file(char *filename) {
+ struct fsm_trie_handle *th;
+ char *text, *textp1, *textp2;
+ int lastword;
+
+ text = file_to_mem(filename);
+ if (text == NULL) {
+ return NULL;
+ }
+ textp1 = text;
+ th = fsm_trie_init();
+
+ for (lastword = 0 ; lastword == 0 ; textp1 = textp2+1) {
+ for (textp2 = textp1 ; *textp2 != '\n' && *textp2 != '\0'; textp2++) {
+ }
+ if (*textp2 == '\0') {
+ lastword = 1;
+ if (textp2 == textp1)
+ break;
+ }
+ *textp2 = '\0';
+ if (strlen(textp1) > 0)
+ fsm_trie_add_word(th, textp1);
+ }
+ xxfree(text);
+ return(fsm_trie_done(th));
+}
+
+int fsm_write_binary_file(struct fsm *net, char *filename) {
+ gzFile outfile;
+ if ((outfile = gzopen(filename,"wb")) == NULL) {
+ return(1);
+ }
+ foma_net_print(net, outfile);
+ gzclose(outfile);
+ return(0);
+}
+
+struct fsm *fsm_read_binary_file_multiple(fsm_read_binary_handle fsrh) {
+ char *net_name;
+ struct fsm *net;
+ struct io_buf_handle *iobh;
+ iobh = (struct io_buf_handle *) fsrh;
+ net = io_net_read(iobh, &net_name);
+ if (net == NULL) {
+ io_free(iobh);
+ return(NULL);
+ } else {
+ xxfree(net_name);
+ return(net);
+ }
+}
+
+fsm_read_binary_handle fsm_read_binary_file_multiple_init(char *filename) {
+
+ struct io_buf_handle *iobh;
+ fsm_read_binary_handle fsm_read_handle;
+
+ iobh = io_init();
+ if (io_gz_file_to_mem(iobh, filename) == 0) {
+ io_free(iobh);
+ return NULL;
+ }
+ fsm_read_handle = (void *) iobh;
+ return(fsm_read_handle);
+}
+
+struct fsm *fsm_read_binary_file(char *filename) {
+ char *net_name;
+ struct fsm *net;
+ struct io_buf_handle *iobh;
+ iobh = io_init();
+ if (io_gz_file_to_mem(iobh, filename) == 0) {
+ io_free(iobh);
+ return NULL;
+ }
+ net = io_net_read(iobh, &net_name);
+ io_free(iobh);
+ return(net);
+}
+
+int save_defined(struct defined_networks *def, char *filename) {
+ struct defined_networks *d;
+ gzFile outfile;
+ if (def == NULL) {
+ fprintf(stderr, "No defined networks.\n");
+ return(0);
+ }
+ if ((outfile = gzopen(filename, "wb")) == NULL) {
+ printf("Error opening file %s for writing.\n", filename);
+ return(-1);
+ }
+ printf("Writing definitions to file %s.\n", filename);
+ for (d = def; d != NULL; d = d->next) {
+ strcpy(d->net->name, d->name);
+ foma_net_print(d->net, outfile);
+ }
+ gzclose(outfile);
+ return(1);
+}
+
+int load_defined(struct defined_networks *def, char *filename) {
+ struct fsm *net;
+ char *net_name;
+ struct io_buf_handle *iobh;
+
+ iobh = io_init();
+ printf("Loading definitions from %s.\n",filename);
+ if (io_gz_file_to_mem(iobh, filename) == 0) {
+ fprintf(stderr, "File error.\n");
+ io_free(iobh);
+ return 0;
+ }
+ while ((net = io_net_read(iobh, &net_name)) != NULL) {
+ add_defined(def, net, net_name);
+ }
+ io_free(iobh);
+ return(1);
+}
+
+static INLINE int explode_line(char *buf, int *values) {
+ int i, j, items;
+ j = i = items = 0;
+ for (;;) {
+ for (i = j; *(buf+j) != ' ' && *(buf+j) != '\0'; j++) { }
+ if (*(buf+j) == '\0') {
+ *(values+items) = atoi(buf+i);
+ items++;
+ break;
+ } else{
+ *(buf+j) = '\0';
+ *(values+items) = atoi(buf+i);
+ items++;
+ j++;
+ }
+ }
+ return(items);
+}
+
+/* The file format we use is an extremely simple text format */
+/* which is gzip compressed through libz and consists of the following sections: */
+
+/* ##foma-net VERSION##*/
+/* ##props## */
+/* PROPERTIES LINE */
+/* ##sigma## */
+/* ...SIGMA LINES... */
+/* ##states## */
+/* ...TRANSITION LINES... */
+/* ##end## */
+
+/* Several networks may be concatenated in one file */
+
+/* The initial identifier is "##foma-net 1.0##" */
+/* where 1.0 is the version number for the file format */
+/* followed by the line "##props##" */
+/* which is followed by a line of space separated integers */
+/* which correpond to: */
+
+/* arity arccount statecount linecount finalcount pathcount is_deterministic */
+/* is_pruned is_minimized is_epsilon_free is_loop_free is_completed name */
+
+/* where name is used if defined networks are saved/loaded */
+
+/* Following the props line, we accept anything (for future expansion) */
+/* until we find ##sigma## */
+
+/* the section beginning with "##sigma##" consists of lines with two fields: */
+/* number string */
+/* correponding to the symbol number and the symbol string */
+
+/* the section beginning with "##states##" consists of lines of ASCII integers */
+/* with 2-5 fields to avoid some redundancy in every line corresponding to a */
+/* transition where otherwise state numbers would be unnecessarily repeated and */
+/* out symbols also (if in = out as is the case for recognizers/simple automata) */
+
+/* The information depending on the number of fields in the lines is as follows: */
+
+/* 2: in target (here state_no is the same as the last mentioned one and out = in) */
+/* 3: in out target (again, state_no is the same as the last mentioned one) */
+/* 4: state_no in target final_state (where out = in) */
+/* 5: state_no in out target final_state */
+
+/* There is no harm in always using 5 fields; however this will take up more space */
+
+/* As in struct fsm_state, states without transitions are represented as a 4-field: */
+/* state_no -1 -1 final_state (since in=out for 4-field lines, out = -1 as well) */
+
+/* AS gzopen will read uncompressed files as well, one can gunzip a file */
+/* that contains a network and still read it */
+
+struct fsm *io_net_read(struct io_buf_handle *iobh, char **net_name) {
+
+ char buf[READ_BUF_SIZE];
+ struct fsm *net;
+ struct fsm_state *fsm;
+
+ char *new_symbol;
+ int i, items, new_symbol_number, laststate, lineint[5], *cm;
+ int extras;
+ char last_final = '1';
+
+ if (io_gets(iobh, buf) == 0) {
+ return NULL;
+ }
+
+ net = fsm_create("");
+
+ if (strcmp(buf, "##foma-net 1.0##") != 0) {
+ fsm_destroy(net);
+ perror("File format error foma!\n");
+ return NULL;
+ }
+ io_gets(iobh, buf);
+ if (strcmp(buf, "##props##") != 0) {
+ perror("File format error props!\n");
+ fsm_destroy(net);
+ return NULL;
+ }
+ /* Properties */
+ io_gets(iobh, buf);
+ extras = 0;
+ sscanf(buf, "%i %i %i %i %i %lld %i %i %i %i %i %i %s", &net->arity, &net->arccount, &net->statecount, &net->linecount, &net->finalcount, &net->pathcount, &net->is_deterministic, &net->is_pruned, &net->is_minimized, &net->is_epsilon_free, &net->is_loop_free, &extras, buf);
+ strcpy(net->name, buf);
+ *net_name = xxstrdup(buf);
+ io_gets(iobh, buf);
+
+ net->is_completed = (extras & 3);
+ net->arcs_sorted_in = (extras & 12) >> 2;
+ net->arcs_sorted_out = (extras & 48) >> 4;
+
+ /* Sigma */
+ while (strcmp(buf, "##sigma##") != 0) { /* Loop until we encounter ##sigma## */
+ if (buf[0] == '\0') {
+ printf("File format error at sigma definition!\n");
+ fsm_destroy(net);
+ return NULL;
+ }
+ io_gets(iobh, buf);
+ }
+
+ for (;;) {
+ io_gets(iobh, buf);
+ if (buf[0] == '#') break;
+ if (buf[0] == '\0') continue;
+ new_symbol = strstr(buf, " ");
+ new_symbol[0] = '\0';
+ new_symbol++;
+ if (new_symbol[0] == '\0') {
+ sscanf(buf,"%i", &new_symbol_number);
+ sigma_add_number(net->sigma, "\n", new_symbol_number);
+ } else {
+ sscanf(buf,"%i", &new_symbol_number);
+ sigma_add_number(net->sigma, new_symbol, new_symbol_number);
+ }
+ }
+
+ /* States */
+ if (strcmp(buf, "##states##") != 0) {
+ printf("File format error!\n");
+ return NULL;
+ }
+ net->states = xxmalloc(net->linecount*sizeof(struct fsm_state));
+ fsm = net->states;
+ laststate = -1;
+ for (i=0; ;i++) {
+ io_gets(iobh, buf);
+ if (buf[0] == '#') break;
+
+ /* scanf is just too slow here */
+
+ //items = sscanf(buf, "%i %i %i %i %i",&lineint[0], &lineint[1], &lineint[2], &lineint[3], &lineint[4]);
+
+ items = explode_line(buf, &lineint[0]);
+
+ switch (items) {
+ case 2:
+ (fsm+i)->state_no = laststate;
+ (fsm+i)->in = lineint[0];
+ (fsm+i)->out = lineint[0];
+ (fsm+i)->target = lineint[1];
+ (fsm+i)->final_state = last_final;
+ break;
+ case 3:
+ (fsm+i)->state_no = laststate;
+ (fsm+i)->in = lineint[0];
+ (fsm+i)->out = lineint[1];
+ (fsm+i)->target = lineint[2];
+ (fsm+i)->final_state = last_final;
+ break;
+ case 4:
+ (fsm+i)->state_no = lineint[0];
+ (fsm+i)->in = lineint[1];
+ (fsm+i)->out = lineint[1];
+ (fsm+i)->target = lineint[2];
+ (fsm+i)->final_state = lineint[3];
+ laststate = lineint[0];
+ last_final = lineint[3];
+ break;
+ case 5:
+ (fsm+i)->state_no = lineint[0];
+ (fsm+i)->in = lineint[1];
+ (fsm+i)->out = lineint[2];
+ (fsm+i)->target = lineint[3];
+ (fsm+i)->final_state = lineint[4];
+ laststate = lineint[0];
+ last_final = lineint[4];
+ break;
+ default:
+ printf("File format error\n");
+ return NULL;
+ }
+ if (laststate > 0) {
+ (fsm+i)->start_state = 0;
+ } else if (laststate == -1) {
+ (fsm+i)->start_state = -1;
+ } else {
+ (fsm+i)->start_state = 1;
+ }
+
+ }
+ if (strcmp(buf, "##cmatrix##") == 0) {
+ cmatrix_init(net);
+ cm = net->medlookup->confusion_matrix;
+ for (;;) {
+ io_gets(iobh, buf);
+ if (buf[0] == '#') break;
+ sscanf(buf,"%i", &i);
+ *cm = i;
+ cm++;
+ }
+ }
+ if (strcmp(buf, "##end##") != 0) {
+ printf("File format error!\n");
+ return NULL;
+ }
+ return(net);
+}
+
+static int io_gets(struct io_buf_handle *iobh, char *target) {
+ int i;
+ for (i = 0; *((iobh->io_buf_ptr)+i) != '\n' && *((iobh->io_buf_ptr)+i) != '\0'; i++) {
+ *(target+i) = *((iobh->io_buf_ptr)+i);
+ }
+ *(target+i) = '\0';
+ if (*((iobh->io_buf_ptr)+i) == '\0')
+ (iobh->io_buf_ptr) = (iobh->io_buf_ptr) + i;
+ else
+ (iobh->io_buf_ptr) = (iobh->io_buf_ptr) + i + 1;
+
+ return(i);
+}
+
+int foma_net_print(struct fsm *net, gzFile outfile) {
+ struct sigma *sigma;
+ struct fsm_state *fsm;
+ int i, maxsigma, laststate, *cm, extras;
+
+ /* Header */
+ gzprintf(outfile, "%s","##foma-net 1.0##\n");
+
+ /* Properties */
+ gzprintf(outfile, "%s","##props##\n");
+
+ extras = (net->is_completed) | (net->arcs_sorted_in << 2) | (net->arcs_sorted_out << 4);
+
+ gzprintf(outfile,
+ "%i %i %i %i %i %lld %i %i %i %i %i %i %s\n", net->arity, net->arccount, net->statecount, net->linecount, net->finalcount, net->pathcount, net->is_deterministic, net->is_pruned, net->is_minimized, net->is_epsilon_free, net->is_loop_free, extras, net->name);
+
+ /* Sigma */
+ gzprintf(outfile, "%s","##sigma##\n");
+ for (sigma = net->sigma; sigma != NULL && sigma->number != -1; sigma = sigma->next) {
+ gzprintf(outfile, "%i %s\n",sigma->number, sigma->symbol);
+ }
+
+ /* State array */
+ laststate = -1;
+ gzprintf(outfile, "%s","##states##\n");
+ for (fsm = net->states; fsm->state_no !=-1; fsm++) {
+ if (fsm->state_no != laststate) {
+ if (fsm->in != fsm->out) {
+ gzprintf(outfile, "%i %i %i %i %i\n",fsm->state_no, fsm->in, fsm->out, fsm->target, fsm->final_state);
+ } else {
+ gzprintf(outfile, "%i %i %i %i\n",fsm->state_no, fsm->in, fsm->target, fsm->final_state);
+ }
+ } else {
+ if (fsm->in != fsm->out) {
+ gzprintf(outfile, "%i %i %i\n", fsm->in, fsm->out, fsm->target);
+ } else {
+ gzprintf(outfile, "%i %i\n", fsm->in, fsm->target);
+ }
+ }
+ laststate = fsm->state_no;
+ }
+ /* Sentinel for states */
+ gzprintf(outfile, "-1 -1 -1 -1 -1\n");
+
+ /* Store confusion matrix */
+ if (net->medlookup != NULL && net->medlookup->confusion_matrix != NULL) {
+
+ gzprintf(outfile, "%s","##cmatrix##\n");
+ cm = net->medlookup->confusion_matrix;
+ maxsigma = sigma_max(net->sigma)+1;
+ for (i=0; i < maxsigma*maxsigma; i++) {
+ gzprintf(outfile, "%i\n", *(cm+i));
+ }
+ }
+
+ /* End */
+ gzprintf(outfile, "%s","##end##\n");
+ return(1);
+}
+
+int net_print_att(struct fsm *net, FILE *outfile) {
+ struct fsm_state *fsm;
+ struct fsm_sigma_list *sl;
+ int i, prev;
+
+ fsm = net->states;
+ sl = sigma_to_list(net->sigma);
+ if (sigma_max(net->sigma) >= 0) {
+ (sl+0)->symbol = g_att_epsilon;
+ }
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1) {
+ fprintf(outfile, "%i\t%i\t%s\t%s\n",(fsm+i)->state_no,(fsm+i)->target, (sl+(fsm+i)->in)->symbol, (sl+(fsm+i)->out)->symbol);
+ }
+ }
+ prev = -1;
+ for (i=0; (fsm+i)->state_no != -1; prev = (fsm+i)->state_no, i++) {
+ if ((fsm+i)->state_no != prev && (fsm+i)->final_state == 1) {
+ fprintf(outfile, "%i\n",(fsm+i)->state_no);
+ }
+ }
+ xxfree(sl);
+ return(1);
+}
+
+static size_t io_get_gz_file_size(char *filename) {
+
+ FILE *infile;
+ size_t numbytes;
+ unsigned char bytes[4];
+ unsigned int ints[4], i;
+
+ /* The last four bytes in a .gz file shows the size of the uncompressed data */
+ infile = fopen(filename, "r");
+ fseek(infile, -4, SEEK_END);
+ fread(&bytes, 1, 4, infile);
+ fclose(infile);
+ for (i = 0 ; i < 4 ; i++) {
+ ints[i] = bytes[i];
+ }
+ numbytes = ints[0] | (ints[1] << 8) | (ints[2] << 16 ) | (ints[3] << 24);
+ return(numbytes);
+}
+
+static size_t io_get_regular_file_size(char *filename) {
+
+ FILE *infile;
+ size_t numbytes;
+
+ infile = fopen(filename, "r");
+ fseek(infile, 0L, SEEK_END);
+ numbytes = ftell(infile);
+ fclose(infile);
+ return(numbytes);
+}
+
+
+static size_t io_get_file_size(char *filename) {
+ gzFile FILE;
+ size_t size;
+ FILE = gzopen(filename, "r");
+ if (FILE == NULL) {
+ return(0);
+ }
+ if (gzdirect(FILE) == 1) {
+ gzclose(FILE);
+ size = io_get_regular_file_size(filename);
+ } else {
+ gzclose(FILE);
+ size = io_get_gz_file_size(filename);
+ }
+ return(size);
+}
+
+size_t io_gz_file_to_mem(struct io_buf_handle *iobh, char *filename) {
+
+ size_t size;
+ gzFile FILE;
+
+ size = io_get_file_size(filename);
+ if (size == 0) {
+ return 0;
+ }
+ (iobh->io_buf) = xxmalloc((size+1)*sizeof(char));
+ FILE = gzopen(filename, "rb");
+ gzread(FILE, iobh->io_buf, size);
+ gzclose(FILE);
+ *((iobh->io_buf)+size) = '\0';
+ iobh->io_buf_ptr = iobh->io_buf;
+ return(size);
+}
+
+char *file_to_mem(char *name) {
+ FILE *infile;
+ size_t numbytes;
+ char *buffer;
+ infile = fopen(name, "r");
+ if(infile == NULL) {
+ printf("Error opening file '%s'\n",name);
+ return NULL;
+ }
+ fseek(infile, 0L, SEEK_END);
+ numbytes = ftell(infile);
+ fseek(infile, 0L, SEEK_SET);
+ buffer = (char*)xxmalloc((numbytes+1) * sizeof(char));
+ if(buffer == NULL) {
+ printf("Error reading file '%s'\n",name);
+ return NULL;
+ }
+ if (fread(buffer, sizeof(char), numbytes, infile) != numbytes) {
+ printf("Error reading file '%s'\n",name);
+ return NULL;
+ }
+ fclose(infile);
+ *(buffer+numbytes)='\0';
+ return(buffer);
+}
diff --git a/lexc.h b/lexc.h
new file mode 100644
index 0000000..d0d2580
--- /dev/null
+++ b/lexc.h
@@ -0,0 +1,11 @@
+void lexc_init();
+void lexc_add_mc(char *symbol);
+int lexc_find_mc(char *symbol);
+struct states *lexc_find_lex_state(char *name);
+void lexc_add_word();
+struct fsm *lexc_to_fsm(void);
+void lexc_set_current_lexicon(char *name, int which);
+void lexc_set_current_word(char *name);
+void lexc_clear_current_word();
+void lexc_set_network(struct fsm *net);
+void lexc_trim(char *s);
diff --git a/lexc.l b/lexc.l
new file mode 100644
index 0000000..7a85f61
--- /dev/null
+++ b/lexc.l
@@ -0,0 +1,245 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+%option noyywrap
+%option nounput
+%option noinput
+%top{
+#define YY_BUF_SIZE 16777216
+}
+%{
+#include <stdio.h>
+#include "foma.h"
+#include "lexc.h"
+
+#define SOURCE_LEXICON 0
+#define TARGET_LEXICON 1
+#define YY_USER_ACTION lexccolumn += lexcleng;
+static int lexentries;
+extern int lexclex();
+static struct defined_networks *olddefines;
+extern int my_yyparse(char *my_string, int lineno, struct defined_networks *defined_nets, struct defined_functions *defined_funcs);
+extern struct fsm *current_parse;
+static char *tempstr;
+int lexccolumn = 0;
+
+struct fsm *fsm_lexc_parse_string(char *string, int verbose) {
+
+ olddefines = g_defines;
+ YY_BUFFER_STATE my_string_buffer;
+ my_string_buffer = lexc_scan_string(string);
+ lexentries = -1;
+ lexclineno = 1;
+ lexc_init();
+ if (lexclex() != 1) {
+ if (lexentries != -1) {
+ printf("%i\n",lexentries);
+ }
+ }
+ lexc_delete_buffer(my_string_buffer);
+ g_defines = olddefines;
+ return(lexc_to_fsm());
+}
+
+struct fsm *fsm_lexc_parse_file(char *filename, int verbose) {
+ char *mystring;
+ mystring = file_to_mem(filename);
+ return(fsm_lexc_parse_string(mystring, verbose));
+}
+
+void lexc_trim(char *s) {
+ /* Remove trailing ; and = and space and initial space */
+ int i,j;
+ for (i = strlen(s)-1; *(s+i) == ';' || *(s+i) == '=' || *(s+i) == ' ' || *(s+i) == '\t'; i--)
+ *(s+i) = '\0';
+ for (i=0; *(s+i) == ' ' || *(s+i) == '\t' || *(s+i) == '\n'; i++) {
+ }
+ for (j=0; *(s+i) != '\0'; i++, j++) {
+ *(s+j) = *(s+i);
+ }
+ *(s+j) = *(s+i);
+}
+
+%}
+
+ /* Nonreserved = anything except ; < > ! or space */
+
+NONRESERVED [\001-\177]{-}[\011\012\014\015\040\041\042\045\073\074\076]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277]|[\045][\001-\177]|[\045][\300-\337][\200-\277]|[\045][\340-\357][\200-\277][\200-\277]|[\045][\360-\367][\200-\277][\200-\277][\200-\277]
+
+INFOSTRING [\001-\177]{-}[\042\012\015]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277]
+
+INSIDEREGEX [\001-\177]{-}[\073\173\175\042\045\076]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277]|(@>)|(>@)|(->)|(=>)
+
+INSIDEDEFREGEX [\001-\177]{-}[\073\173\175\042\045]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277]
+
+SPACE [\040]|[\011]|[\014]
+
+ANY [\001-\177]|[\300-\337][\200-\277]|[\340-\357][\200-\277][\200-\277]|[\360-\367][\200-\277][\200-\277][\200-\277]
+
+%x MCS LEXICON DEF LEXENTRIES INSIDEREGEX REGEX REGEXB REGEXQ DEFREGEX DEFREGEXB DEFREGEXQ EATUPINFO
+%%
+
+ /* Files begin with one of these three identifiers */
+<*>Multichar_Symbols {
+ BEGIN(MCS);
+}
+
+<*>Definitions {
+ BEGIN(DEF);
+}
+
+ /* This line needs to be above the space glob */
+ /* otherwise spaces get eaten up in a regex */
+<REGEX>({INSIDEREGEX}|%{ANY})* {
+ yymore();
+}
+
+<*>{SPACE}+ { }
+<*>[\015]?\n { lexclineno++; lexccolumn = 1;}
+ /* Multichar definitions */
+
+ /* A Multichar definition can contain anything except nonescaped space */
+<MCS>{NONRESERVED}+ {
+ lexc_add_mc(lexctext);
+}
+
+<*>(LEXICON|Lexicon){SPACE}+{NONRESERVED}+ {
+ lexc_trim(lexctext+8);
+ if (lexentries != -1) {
+ printf("%i, ",lexentries);
+ }
+ printf("%s...",lexctext+8);
+ fflush(stdout);
+ lexentries = 0;
+ lexc_set_current_lexicon(lexctext+8, SOURCE_LEXICON);
+ BEGIN(LEXENTRIES);
+}
+
+ /* Grab info string */
+<EATUPINFO>[\042]{INFOSTRING}*[\042]{SPACE}*; {
+ BEGIN(LEXENTRIES);
+}
+ /* Target followed by info string */
+<LEXENTRIES>{NONRESERVED}+{SPACE}+/[\042]{INFOSTRING}*[\042]{SPACE}*; {
+ lexc_trim(lexctext);
+ lexc_set_current_lexicon(lexctext, TARGET_LEXICON);
+ lexc_add_word();
+ lexc_clear_current_word();
+ lexentries++;
+ if (lexentries %10000 == 0) {
+ printf("%i...",lexentries);
+ fflush(stdout);
+ }
+ BEGIN(EATUPINFO);
+}
+
+
+ /* Regular entries contain anything (not starting with <) and end in a nonescaped SPACE */
+<LEXENTRIES>{NONRESERVED}+ {
+ lexc_set_current_word(lexctext);
+}
+
+
+<LEXENTRIES>{NONRESERVED}+{SPACE}*; {
+ //printf("[%s]\n", lexctext);
+ lexc_trim(lexctext);
+ lexc_set_current_lexicon(lexctext, TARGET_LEXICON);
+ lexc_add_word();
+ lexc_clear_current_word();
+ lexentries++;
+ if (lexentries %10000 == 0) {
+ printf("%i...",lexentries);
+ fflush(stdout);
+ }
+}
+
+ /* A REGEX entry begins and ends with a < , > */
+<LEXENTRIES>[\074] {
+ BEGIN(REGEX);
+}
+ /* \076 = > */
+<REGEX>[\076] {
+ *(lexctext+lexcleng-1) = ';';
+ if (my_yyparse(lexctext, lexclineno, g_defines, NULL) == 0) {
+ lexc_set_network(current_parse);
+ }
+ BEGIN(LEXENTRIES);
+}
+
+<REGEX>[{] {
+ BEGIN(REGEXB);
+ yymore();
+}
+<REGEXB>[^}] {
+ yymore();
+}
+<REGEXB>[}] {
+ BEGIN(REGEX);
+ yymore();
+}
+<REGEX>(["])* {
+ BEGIN(REGEXQ);
+ yymore();
+}
+<REGEXQ>([^"]*) {
+ yymore();
+}
+<REGEXQ>([\042]) {
+ BEGIN(REGEX);
+ yymore();
+}
+<DEF>{NONRESERVED}+{SPACE}+={SPACE}+ {
+ lexc_trim(lexctext);
+ tempstr = xxstrdup(lexctext);
+ BEGIN(DEFREGEX);
+}
+ /* \073 = ; */
+<DEFREGEX>[\073] {
+ if (my_yyparse(lexctext, lexclineno, g_defines, NULL) == 0) {
+ add_defined(g_defines, fsm_topsort(fsm_minimize(current_parse)),tempstr);
+ }
+ xxfree(tempstr);
+ BEGIN(DEF);
+}
+<DEFREGEX>({INSIDEDEFREGEX}|%{ANY})* {
+ yymore();
+}
+<DEFREGEX>[{] {
+ BEGIN(DEFREGEXB);
+ yymore();
+}
+<DEFREGEXB>[^}] {
+ yymore();
+}
+<DEFREGEXB>[}] {
+ BEGIN(DEFREGEX);
+ yymore();
+}
+<DEFREGEX>(["])* {
+ BEGIN(DEFREGEXQ);
+ yymore();
+}
+<DEFREGEXQ>([^"]*) {
+ yymore();
+}
+<DEFREGEXQ>([\042]) {
+ BEGIN(DEFREGEX);
+ yymore();
+}
+<*>((!).*[\015]?(\n)) { /* printf ("Comment: [%s]\n",lexctext); */ }
+
+<*>(.) { printf("\n***Syntax error on line %i column %i at '%s'\n",lexclineno,lexccolumn,lexctext); return 1;}
diff --git a/lexcread.c b/lexcread.c
new file mode 100644
index 0000000..dfdfe4b
--- /dev/null
+++ b/lexcread.c
@@ -0,0 +1,1087 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "foma.h"
+#include "lexc.h"
+
+#define SIGMA_HASH_TABLESIZE 3079
+
+#define WORD_ENTRY 1
+#define REGEX_ENTRY 2
+
+extern int g_lexc_align;
+
+struct multichar_symbols {
+ char *symbol;
+ short int sigma_number;
+ struct multichar_symbols *next;
+};
+
+struct lexstates { /* Separate list of LEXICON states */
+ char *name;
+ struct states *state;
+ struct lexstates *next;
+ unsigned char targeted;
+ unsigned char has_outgoing;
+};
+
+struct states {
+ struct trans {
+ short int in;
+ short int out;
+ struct states *target;
+ struct trans *next;
+ } *trans;
+ struct lexstates *lexstate; /* ptr to lexicon state */
+ int number; /* State number (generated later) */
+ unsigned int hashval; /* Hash for remaining symbols until next lexstate */
+ unsigned char mergeable; /* Can this state be merged with other suffix */
+ /* 0 = NO, 1 = YES, 2 = DELETED/MERGED */
+ unsigned short int distance; /* Number of remaining symbols until lexstate */
+ struct states *merge_with;
+};
+
+struct statelist {
+ struct states *state;
+ struct statelist *next;
+ char start;
+ char final;
+};
+
+struct lexc_hashtable { /* Hash for looking up symbols in sigma quickly */
+ char *symbol;
+ struct lexc_hashtable *next;
+ int sigma_number;
+};
+
+static unsigned int primes[26] = {61,127,251,509,1021,2039,4093,8191,16381,32749,65521,131071,262139,524287,1048573,2097143,4194301,8388593,16777213,33554393,67108859,134217689,268435399,536870909,1073741789,2147483647};
+
+static struct statelist *statelist = NULL;
+static struct multichar_symbols *mc = NULL;
+static struct lexstates *lexstates = NULL;
+static struct sigma *lexsigma = NULL;
+static struct lexc_hashtable *hashtable;
+static struct fsm *current_regex_network;
+
+static int cwordin[1000], cwordout[1000], medcwordin[2000], medcwordout[2000], carity, lexc_statecount, maxlen, hasfinal, current_entry, net_has_unknown;
+static _Bool *mchash;
+static struct lexstates *clexicon, *ctarget;
+
+static char *mystrncpy(char *dest, char *src, int len);
+static void lexc_string_to_tokens(char *string, int *intarr);
+static void lexc_pad();
+static void lexc_medpad();
+static void lexc_number_states();
+static void lexc_cleanup();
+static unsigned int lexc_suffix_hash(int offset);
+static unsigned int lexc_symbol_hash(char *s);
+static void lexc_update_unknowns(int sigma_number);
+
+static unsigned int lexc_suffix_hash(int offset) {
+ register unsigned int h = 0, g, p;
+ /* Read suffixes in cwordin[] and cwordout[] and return a hash value */
+ for(p = offset; cwordin[p] != -1; p++) {
+ h = (h << 4) + (unsigned int) (cwordin[p] | (cwordout[p] << 8));
+ if (0 != (g = h & 0xf0000000)) {
+ h = h ^ (g >> 24);
+ h = h ^ g;
+ }
+ }
+ /* No tablemod here, we decide on the table size later */
+ return h;
+}
+
+static unsigned int lexc_symbol_hash(char *s) {
+ register unsigned int hash;
+ int c;
+ hash = 5381;
+ while ((c = *s++))
+ hash = ((hash << 5) + hash) + c;
+ return (hash % SIGMA_HASH_TABLESIZE);
+}
+
+int lexc_find_sigma_hash(char *symbol) {
+ int ptr;
+ struct lexc_hashtable *h;
+ ptr = lexc_symbol_hash(symbol);
+
+ if ((hashtable+ptr)->symbol == NULL)
+ return -1;
+ for (h = (hashtable+ptr); h != NULL; h = h->next) {
+ if (strcmp(symbol,h->symbol) == 0) {
+ return (h->sigma_number);
+ }
+ }
+ return -1;
+}
+
+void lexc_add_sigma_hash(char *symbol, int number) {
+ int ptr;
+ struct lexc_hashtable *h, *hnew;
+ ptr = lexc_symbol_hash(symbol);
+
+ if (net_has_unknown == 1)
+ lexc_update_unknowns(number);
+
+ if ((hashtable+ptr)->symbol == NULL) {
+ (hashtable+ptr)->symbol = xxstrdup(symbol);
+ (hashtable+ptr)->sigma_number = number;
+ return;
+ }
+ for (h = hashtable+ptr; h->next != NULL; h = h->next) {
+ }
+ hnew = xxmalloc(sizeof(struct lexc_hashtable));
+ hnew->symbol = xxstrdup(symbol);
+ hnew->sigma_number = number;
+ h->next = hnew;
+ hnew->next = NULL;
+}
+
+void lexc_init() {
+ int i;
+ lexsigma = sigma_create();
+ mc = NULL;
+ lexstates = NULL;
+ clexicon = NULL;
+ ctarget = NULL;
+ statelist = NULL;
+ lexc_statecount = 0;
+ net_has_unknown = 0;
+ lexc_clear_current_word();
+ hashtable = xxcalloc(SIGMA_HASH_TABLESIZE, sizeof(struct lexc_hashtable));
+
+ maxlen = 0;
+
+ mchash = xxcalloc(256*256, sizeof(_Bool));
+ for (i=0; i< SIGMA_HASH_TABLESIZE; i++) {
+ (hashtable+i)->symbol = NULL;
+ (hashtable+i)->sigma_number = -1;
+ (hashtable+i)->next = NULL;
+ }
+}
+
+void lexc_clear_current_word() {
+ cwordin[0] = cwordout[0] = 0;
+ cwordin[1] = cwordout[1] = -1;
+ current_entry = WORD_ENTRY;
+}
+
+void lexc_add_state(struct states *s) {
+ struct statelist *sl;
+ sl = xxmalloc(sizeof(struct statelist));
+ sl->state = s;
+ s->number = -1;
+ sl->next = statelist;
+ sl->start = 0;
+ sl->final = 0;
+ statelist = sl;
+ lexc_statecount++;
+}
+
+/* Go through the net built so far and add new transitions for @ */
+/* to reflect the new symbols we now have in sigma */
+/* We should really build a fast lookup ptr for finding the @ transitions */
+/* But who in their right mind is ever going to use lots of @ in a lexicon construction? */
+/* Of course this only applies to the special construct < regex > inside lexicon entries */
+/* since @ is impossible to produce otherwise */
+
+void lexc_update_unknowns(int sigma_number) {
+ struct statelist *s;
+ struct trans *t, *newtrans;
+ for (s = statelist; s != NULL; s = s->next) {
+ if (s->state->mergeable == 2)
+ continue;
+ for (t=s->state->trans ; t!=NULL; t= t->next) {
+ if (t->in == IDENTITY || t->out == IDENTITY) {
+ newtrans = xxmalloc(sizeof(struct trans));
+ newtrans->in = sigma_number;
+ newtrans->out = sigma_number;
+ newtrans->target = t->target;
+ newtrans->next = t->next;
+ t->next = newtrans;
+ }
+ }
+ }
+}
+
+void lexc_add_network() {
+
+ struct fsm *net;
+ struct fsm_state *fsm;
+ struct sigma *sigma;
+ struct states **slist, *sourcestate, *deststate, *newstate;
+ struct statelist *s;
+ struct trans *newtrans;
+ int i, j, *sigreplace, signumber, maxstate, *finals, unknown_symbols, first_new_sigma, *unk = NULL;
+
+ unknown_symbols = 0;
+ first_new_sigma = 0;
+ sourcestate = clexicon->state;
+ deststate = ctarget->state;
+
+ net = current_regex_network;
+ fsm = net->states;
+
+ sigreplace = xxcalloc(sigma_max(net->sigma)+1,sizeof(int));
+
+ for (sigma = net->sigma; sigma != NULL && sigma->number != -1; sigma = sigma->next) {
+ if ((signumber = lexc_find_sigma_hash(sigma->symbol)) == -1) {
+ /* Add to existing lexc sigma */
+ signumber = sigma_add(sigma->symbol, lexsigma);
+ first_new_sigma = first_new_sigma > 0 ? first_new_sigma : signumber;
+ lexc_add_sigma_hash(sigma->symbol, signumber);
+ *(sigreplace+sigma->number) = signumber;
+ } else {
+ /* We already have it, add to conversion table */
+ *(sigreplace+sigma->number) = signumber;
+ }
+ }
+
+ /* Renum arcs */
+ for (i=0, maxstate = 0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->in != -1)
+ (fsm+i)->in = *(sigreplace+(fsm+i)->in);
+ if ((fsm+i)->out != -1)
+ (fsm+i)->out = *(sigreplace+(fsm+i)->out);
+ maxstate = (fsm+i)->state_no > maxstate ? (fsm+i)->state_no : maxstate;
+ if ((fsm+i)->in == IDENTITY || (fsm+i)->in == UNKNOWN || (fsm+i)->out == UNKNOWN)
+ unknown_symbols = 1;
+ }
+ if (unknown_symbols == 1) {
+ unk = xxcalloc(sigma_max(lexsigma)+2,sizeof(int));
+ for (i=0, sigma = lexsigma; sigma != NULL && sigma->number != -1; sigma=sigma->next) {
+ if (sigma->number > 2 && sigma_find(sigma->symbol, net->sigma) == -1) {
+ *(unk+i) = sigma->number;
+ i++;
+ }
+ }
+ }
+
+ slist = xxcalloc(sizeof(**slist),maxstate+1);
+ finals = xxcalloc(sizeof(int),maxstate+1);
+
+ for (i=0; i <= maxstate;i++) {
+ newstate = xxmalloc(sizeof(struct states));
+ *(slist+i) = newstate;
+ newstate->trans = NULL;
+ newstate->lexstate = NULL;
+ newstate->number = -1;
+ newstate->hashval = -1;
+ newstate->mergeable = 0;
+ newstate->distance = 0;
+ newstate->merge_with = newstate;
+ s = xxmalloc(sizeof(struct statelist));
+ s->state = newstate;
+ s->next = statelist;
+ s->start = 0;
+ s->final = 0;
+ statelist = s;
+ }
+ /* Add an EPSILON transition from sourcestate to state 0 */
+ newtrans = xxmalloc(sizeof(struct trans));
+ newtrans->in = EPSILON;
+ newtrans->out = EPSILON;
+ newtrans->target = *slist;
+ newtrans->next = sourcestate->trans;
+ sourcestate->trans = newtrans;
+
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1) {
+ newstate = *(slist+(fsm+i)->state_no);
+ newtrans = xxmalloc(sizeof(struct trans));
+ newtrans->in = (fsm+i)->in;
+ newtrans->out = (fsm+i)->out;
+ newtrans->target = *(slist+(fsm+i)->target);
+ newtrans->next = newstate->trans;
+ newstate->trans = newtrans;
+ /* Add new symbols for @:@ transitions */
+ /* TODO: make this work for ?: or :? trans as well */
+ if (unknown_symbols == 1) {
+ if ((fsm+i)->in == IDENTITY || (fsm+i)->out == IDENTITY) {
+ for (j=0; *(unk+j) != 0; j++) {
+ newtrans = xxmalloc(sizeof(struct trans));
+ newtrans->in = *(unk+j);
+ newtrans->out = *(unk+j);
+ newtrans->target = *(slist+(fsm+i)->target);
+ newtrans->next = newstate->trans;
+ newstate->trans = newtrans;
+ }
+ }
+ }
+ }
+ finals[(fsm+i)->state_no] = (fsm+i)->final_state;
+ }
+ /* Add an EPSILON transition from all final states to deststate */
+ for (i=0; i <= maxstate; i++) {
+ if (finals[i] == 1) {
+ newtrans = xxmalloc(sizeof(struct trans));
+ newtrans->in = newtrans->out = EPSILON;
+ newtrans->target = deststate;
+ newstate = *(slist+i);
+ newtrans->next = newstate->trans;
+ newstate->trans = newtrans;
+ }
+ }
+ if (unknown_symbols == 1) {
+ xxfree(unk);
+ net_has_unknown = 1;
+ }
+ xxfree(slist);
+ xxfree(finals);
+}
+
+void lexc_set_network(struct fsm *net) {
+ current_regex_network = net;
+ current_entry = REGEX_ENTRY;
+ return;
+}
+
+void lexc_set_current_lexicon(char *name, int which) {
+ /* Sets the global lexicon variable to point to a new lexicon */
+ /* the variable which = 0 indicates source, which = 1 indicated target */
+
+ struct lexstates *l;
+ struct states *newstate;
+
+ for (l = lexstates; l != NULL; l = l->next) {
+ if (strcmp(name,l->name) == 0) {
+ if (which == 0) {
+ l->has_outgoing = 1;
+ clexicon = l;
+ } else {
+ ctarget = l;
+ }
+ return;
+ }
+ }
+ l = xxmalloc(sizeof(struct lexstates));
+ l->next = lexstates;
+ l->name = xxstrdup(name);
+ l->has_outgoing = 0;
+ l->targeted = 0;
+ lexstates = l;
+ newstate = xxmalloc(sizeof(struct states));
+ lexc_add_state(newstate);
+ newstate->lexstate = l;
+ newstate->trans = NULL;
+ newstate->mergeable = 0;
+ newstate->merge_with = newstate;
+ l->state = newstate;
+ if (which == 0) {
+ clexicon = l;
+ l->has_outgoing = 1;
+ } else {
+ ctarget = l;
+ }
+}
+
+char *lexc_find_delim(char *name, char delimiter, char escape) {
+ int i;
+ for (i=0; *(name+i) != '\0'; i++) {
+ if (*(name+i) == escape && *(name+i+1) != '\0') {
+ i++;
+ continue;
+ }
+ if (*(name+i) == delimiter) {
+ return name+i;
+ }
+ }
+ return NULL;
+}
+
+void lexc_deescape_string(char *name, char escape, int mode) {
+ int i, j;
+ for (i=0, j=0; *(name+i) != '\0'; i++) {
+ *(name+j) = *(name+i);
+ if (*(name+i) == escape) {
+ *(name+j) = *(name+i+1);
+ j++;
+ i++;
+ continue;
+ }
+ else if (mode == 1 && *(name+i) == '0') {
+ /* Marks alignment EPSILON */
+ *(name+j) = (unsigned char) 0xff;
+ j++;
+ continue;
+ }
+ else if (*(name+i) != escape && *(name+i) != '0') {
+ j++;
+ continue;
+ }
+ }
+ *(name+j) = '\0';
+}
+
+/* Read a string and fill cwordin, cwordout arrays */
+/* with the sigma numbers of the current word, -1 terminated */
+
+void lexc_set_current_word(char *name) {
+ char *instring, *outstring;
+ int i;
+
+ carity = 1;
+ instring = name;
+ outstring = lexc_find_delim(name,':','%');
+ /* printf("CWin: [%s] CWout: [%s]\n", instring, outstring); */
+ if (outstring != NULL) {
+ *outstring = '\0';
+ outstring = outstring+1;
+ lexc_deescape_string(outstring,'%',1);
+ carity = 2;
+ }
+ lexc_deescape_string(instring, '%',1);
+ /* printf("CWin2: [%s] CWout2: [%s]\n", instring, outstring); */
+
+ lexc_string_to_tokens(instring, cwordin);
+
+ if (carity == 2) {
+ lexc_string_to_tokens(outstring, cwordout);
+ if (g_lexc_align)
+ lexc_medpad();
+ else
+ lexc_pad();
+ } else {
+ for (i=0; *(cwordin+i) != -1; i++) {
+ *(cwordout+i) = *(cwordin+i);
+ }
+ *(cwordout+i) = -1;
+
+ }
+ current_entry = WORD_ENTRY;
+}
+
+
+#define LEV_DOWN 0
+#define LEV_LEFT 1
+#define LEV_DIAG 2
+
+void lexc_medpad() {
+ int i, j, x, y, s1len, s2len, left, down, diag, dir;
+
+ if (*cwordin == -1 && *cwordout == -1) {
+ *cwordin = *cwordout = EPSILON;
+ *(cwordin+1) = *(cwordout+1) = -1;
+ return;
+ }
+
+ for (i = 0, j = 0; cwordin[i] != -1; i++) {
+ if (cwordin[i] == EPSILON) {
+ continue;
+ }
+ cwordin[j] = cwordin[i];
+ j++;
+ }
+ cwordin[j] = -1;
+
+ for (i = 0, j = 0; cwordout[i] != -1; i++) {
+ if (cwordout[i] == EPSILON) {
+ continue;
+ }
+ cwordout[j] = cwordout[i];
+ j++;
+ }
+ cwordout[j] = -1;
+
+ for (i = 0; cwordin[i] != -1; i++) { }
+ s1len = i;
+ for (i = 0; cwordout[i] != -1; i++) { }
+ s2len = i;
+
+ int matrix[s1len+2][s2len+2];
+ int dirmatrix[s1len+2][s2len+2];
+
+ matrix[0][0] = 0;
+ dirmatrix[0][0] = 0;
+ for (x = 1; x <= s1len; x++) {
+ matrix[x][0] = matrix[x-1][0] + 1;
+ dirmatrix[x][0] = LEV_LEFT;
+ }
+ for (y = 1; y <= s2len; y++) {
+ matrix[0][y] = matrix[0][y-1] + 1;
+ dirmatrix[0][y] = LEV_DOWN;
+ }
+ for (x = 1; x <= s1len; x++) {
+ for (y = 1; y <= s2len; y++) {
+ diag = matrix[x-1][y-1] + (cwordin[x-1] == cwordout[y-1] ? 0 : 100);
+ down = matrix[x][y-1] + 1;
+ left = matrix[x-1][y] + 1;
+ if (diag <= left && diag <= down) {
+ matrix[x][y] = diag;
+ dirmatrix[x][y] = LEV_DIAG;
+ } else if (left <= diag && left <= down) {
+ matrix[x][y] = left;
+ dirmatrix[x][y] = LEV_LEFT;
+ } else {
+ matrix[x][y] = down ;
+ dirmatrix[x][y] = LEV_DOWN;
+ }
+ }
+ }
+
+ for (x = s1len, y = s2len, i = 0; (x > 0) || (y > 0); i++) {
+ dir = dirmatrix[x][y];
+ if (dir == LEV_DIAG) {
+ medcwordin[i] = cwordin[x-1];
+ medcwordout[i] = cwordout[y-1];
+ x--;
+ y--;
+ }
+ else if (dir == LEV_DOWN) {
+ medcwordin[i] = EPSILON;
+ medcwordout[i] = cwordout[y-1];
+ y--;
+ }
+ else {
+ medcwordin[i] = cwordin[x-1];
+ medcwordout[i] = EPSILON;
+ x--;
+ }
+ }
+ for (j = 0, i-= 1; i >= 0; j++, i--) {
+ cwordin[j] = medcwordin[i];
+ cwordout[j] = medcwordout[i];
+ }
+ cwordin[j] = -1;
+ cwordout[j] = -1;
+}
+
+void lexc_pad() {
+ int i, pad;
+ /* Pad the shorter of current in, out words in cwordin, cwordout with EPSILON */
+
+ if (*cwordin == -1 && *cwordout == -1) {
+ *cwordin = *cwordout = EPSILON;
+ *(cwordin+1) = *(cwordout+1) = -1;
+ return;
+ }
+
+ for (i=0, pad = 0; ;i++) {
+ if (pad == 1 && *(cwordout+i) == -1) {
+ *(cwordin+i) = -1;
+ break;
+ }
+ if (pad == 2 && *(cwordin+i) == -1) {
+ *(cwordout+i) = -1;
+ break;
+ }
+ if (*(cwordin+i) == -1 && *(cwordout+i) != -1) {
+ pad = 1; /* Pad upper */
+ }
+ else if (*(cwordin+i) != -1 && *(cwordout+i) == -1) {
+ pad = 2; /* Pad lower */
+ }
+ if (pad == 1) {
+ *(cwordin+i) = EPSILON;
+ }
+ if (pad == 2) {
+ *(cwordout+i) = EPSILON;
+ }
+ if (pad == 0 && *(cwordin+i) == -1)
+ break;
+ }
+}
+
+void lexc_string_to_tokens(char *string, int *intarr) {
+ int len, i, pos, skip, signumber, multi;
+ unsigned int mchashval;
+ char tmpstring[5];
+ struct multichar_symbols *mcs;
+ len = strlen(string);
+ for (i=0, pos = 0; i < len; ) {
+
+ /* EPSILON for alignment is marked as 0xff */
+ if ((unsigned char) string[i] == 0xff) {
+ *(intarr+pos) = EPSILON;
+ pos++;
+ i++;
+ continue;
+ }
+
+ multi = 0;
+ mchashval = (unsigned int) ((unsigned char) *(string+i)) * 256 + (unsigned int) ((unsigned char) *(string+i+1));
+ if ((i < len-1) && *(mchash+mchashval) == 1) {
+ for (mcs = mc; mcs != NULL; mcs = mcs->next) {
+ if (strncmp(string+i,mcs->symbol,strlen(mcs->symbol)) == 0) {
+ /* printf("Found multichar: [%s][%i]\n",mcs->symbol,mcs->sigma_number); */
+ multi = 1;
+ break;
+ }
+ }
+ }
+
+ if (multi) {
+ *(intarr+pos) = mcs->sigma_number;
+ pos++;
+ i += strlen(mcs->symbol);
+ } else {
+ skip = utf8skip(string+i);
+ if ((signumber = lexc_find_sigma_hash(mystrncpy(tmpstring,string+i,skip+1))) != -1) {
+ *(intarr+pos) = signumber;
+ pos++;
+ i = i + skip + 1;
+ } else {
+ signumber = sigma_add(mystrncpy(tmpstring, string+i, skip+1), lexsigma);
+ lexc_add_sigma_hash(tmpstring, signumber);
+ *(intarr+pos) = signumber;
+ pos++;
+ i = i + skip + 1;
+ }
+ }
+ }
+ *(intarr+pos) = -1;
+}
+
+char *mystrncpy(char *dest, char *src, int len) {
+ int i;
+ for (i=0; i < len; i++) {
+ *(dest+i) = *(src+i);
+ if (*(src+i) == '\0')
+ return(dest);
+ }
+ *(dest+i) = '\0';
+/* printf("Mystrncpy: [%s]\n",dest); */
+ return(dest);
+}
+
+/* Add MC to front of chain */
+/* In decreasing order of length */
+
+void lexc_add_mc(char *symbol) {
+ int s, len;
+ unsigned int mchashval;
+ struct multichar_symbols *mcs, *mcprev, *mcnew;
+ lexc_deescape_string(symbol,'%',0);
+ if (!lexc_find_mc(symbol)) {
+ len = utf8strlen(symbol);
+ mcprev = NULL;
+ for (mcs = mc; mcs != NULL && utf8strlen(mcs->symbol) > len; mcprev = mcs, mcs=mcs->next) {
+ }
+ mcnew = xxmalloc(sizeof(struct multichar_symbols));
+ mcnew->symbol = xxstrdup(symbol);
+ mcnew->next = mcs;
+ if ((mc == NULL) ||(mcs != NULL && mcprev == NULL))
+ mc = mcnew;
+ if (mcprev != NULL)
+ mcprev->next = mcnew;
+
+ s = sigma_add(symbol, lexsigma);
+ mchashval = (unsigned int) ((unsigned char) *(symbol)) * 256 + (unsigned int) ((unsigned char) *(symbol+1));
+ lexc_add_sigma_hash(symbol, s);
+ *(mchash+mchashval) = 1;
+ mcnew->sigma_number = s;
+ }
+}
+
+int lexc_find_mc(char *symbol) {
+ struct multichar_symbols *mcs;
+ for (mcs = mc ; mcs != NULL ; mcs = mcs->next) {
+ if (strcmp(symbol,mcs->symbol) == 0)
+ return 1;
+ }
+ return 0;
+}
+
+struct states *lexc_find_lex_state(char *name) {
+ struct lexstates *l;
+ for (l = lexstates ; l != NULL; l = l->next) {
+ if (strcmp(name,l->name) == 0)
+ return (l->state);
+ }
+ return NULL;
+}
+
+void lexc_add_word() {
+ /** Add a word from source state to destination state */
+ struct trans *newtrans, *trans;
+ struct states *sourcestate, *deststate, *newstate;
+ int i, follow, len;
+
+ if (current_entry == REGEX_ENTRY) {
+ lexc_add_network();
+ return;
+ }
+
+ /* find source, dest */
+ sourcestate = clexicon->state;
+ deststate = ctarget->state;
+
+ for (i=0; *(cwordin+i) != -1; i++) {}
+ len = i;
+ maxlen = len > maxlen ? len : maxlen;
+
+ /* We follow the source state if the symbols are the same */
+ /* To merge prefixes */
+ for (follow = 1, i=0; *(cwordin+i) != -1; i++) {
+
+ if (follow == 1) {
+ for (trans = sourcestate->trans; trans != NULL ; trans = trans->next) {
+ if (trans->in == *(cwordin+i) && trans->out == *(cwordout+i) && trans->target->lexstate == NULL) {
+ /* Can't follow if target needs to be lexstate */
+ if (*(cwordin+i+1) == -1 && trans->target != deststate) {
+ continue;
+ }
+ sourcestate = trans->target;
+ sourcestate->mergeable = 0;
+ /* Breakout */
+ goto breakout;
+ }
+ }
+ }
+ follow = 0;
+
+ newtrans = xxmalloc(sizeof(struct trans));
+ if (*(cwordin+i+1) == -1) {
+ newtrans->target = deststate;
+ } else {
+ newstate = xxmalloc(sizeof(struct states));
+ lexc_add_state(newstate);
+ newtrans->target = newstate;
+ newstate->trans = NULL;
+ newstate->lexstate = NULL;
+ newstate->mergeable = 1;
+ newstate->hashval = lexc_suffix_hash(i+1);
+ newstate->distance = len - i - 1;
+ newstate->merge_with = newstate;
+ }
+ newtrans->next = sourcestate->trans;
+ sourcestate->trans = newtrans;
+
+ newtrans->in = *(cwordin+i);
+ newtrans->out = *(cwordout+i);
+
+ sourcestate = newtrans->target;
+ breakout:;
+
+ }
+ return;
+}
+
+void lexc_number_states() {
+ int n, smax, hasroot;
+ struct statelist *s;
+ struct lexstates *l;
+
+ smax = n = hasfinal = 0;
+
+ for (hasroot = 0, s = statelist; s != NULL; s = s->next) {
+ smax++;
+ if (s->state->lexstate != NULL && strcmp(s->state->lexstate->name, "Root") == 0) {
+ s->state->number = 0;
+ s->start = 1;
+ n++;
+ hasroot = 1;
+ break;
+ }
+ }
+ /* If there is no Root lexicon, the first lexicon mentioned is Root */
+ if (!hasroot) {
+ for (s = statelist; s != NULL; s = s->next) {
+ if (s->next == NULL) {
+ s->state->number = 0;
+ fprintf(stderr,"*Warning: no Root lexicon, using '%s' as Root.\n",s->state->lexstate->name);
+ s->start = 1;
+ n++;
+ }
+ }
+ }
+ /* Mark # as the last state */
+ for (s = statelist; s != NULL; s = s->next) {
+ if (s->state->lexstate != NULL && strcmp(s->state->lexstate->name, "#") == 0) {
+ s->state->number = smax-1;
+ s->final = 1;
+ hasfinal = 1;
+ } else if (s->state->lexstate != NULL && strcmp(s->state->lexstate->name, "#") != 0 && s->state->lexstate->has_outgoing == 0) {
+ /* Also mark uncontinued states as final (this is warned about elsewhere) */
+ s->final = 1;
+ }
+ }
+
+ for (s = statelist; s != NULL; s = s->next) {
+ if (s->state->number == -1) {
+ s->state->number = n;
+ n++;
+ }
+ }
+ lexc_statecount = n+1;
+ for (l = lexstates; l != NULL ; l = l->next) {
+ if (l->targeted == 0 && l->state->number != 0) {
+ fprintf(stderr,"*Warning: lexicon '%s' defined but not used\n",l->name);
+ fflush(stdout);
+ }
+ if (l->has_outgoing == 0 && strcmp(l->name, "#") != 0) {
+ fprintf(stderr,"***Warning: lexicon '%s' used but never defined\n",l->name);
+ fflush(stdout);
+ }
+ }
+}
+
+int lexc_eq_paths(struct states *one, struct states *two) {
+ while (one->lexstate == NULL && two->lexstate == NULL) {
+ if (one->trans->in != two->trans->in || one->trans->out != two->trans->out)
+ return 0;
+ one = one->trans->target;
+ two = two->trans->target;
+ }
+ if (one->lexstate != two->lexstate)
+ return 0;
+ return 1;
+}
+
+void lexc_merge_states() {
+ struct lenlist {
+ struct states *state;
+ struct lenlist *next;
+ };
+ struct hashstates {
+ struct states *state;
+ struct hashstates *next;
+ } *hashstates, *currenth, *newh;
+
+ struct lenlist *lenlist, *newl, *currentl;
+ struct statelist *s, *sprev, *sf;
+ struct states *state, *purgestate;
+ struct trans *t, *tprev;
+ int i, numstates, tablesize, hash;
+
+ /* Create array of ptrs to states depending on string length */
+ lenlist = xxcalloc(maxlen+1,sizeof(struct lenlist));
+ numstates = 0;
+ for (s = statelist ; s!= NULL; s = s->next) {
+ if (s->state->mergeable)
+ numstates++;
+ }
+
+ /* Find a suitable prime for hashing: proportional to the size of the */
+ /* number of mergeable states */
+
+ for (i = 0; primes[i] < numstates/4; i++) { }
+ tablesize = primes[i];
+ hashstates = xxcalloc(tablesize,sizeof(struct hashstates));
+
+ for (s = statelist ; s!= NULL; s = s->next) {
+ if (s->state->mergeable) {
+ numstates++;
+ currentl = lenlist+(s->state->distance);
+ if (currentl->state == NULL)
+ currentl->state = s->state;
+ else {
+ newl = xxcalloc(1,sizeof(struct lenlist));
+ newl->state = s->state;
+ newl->next = currentl->next;
+ currentl->next = newl;
+ }
+ s->state->hashval = s->state->hashval % tablesize;
+ currenth = hashstates+s->state->hashval;
+ if (currenth->state == NULL) {
+ currenth->state = s->state;
+ } else {
+ newh = xxcalloc(1,sizeof(struct hashstates));
+ newh->state = s->state;
+ newh->next = currenth->next;
+ currenth->next = newh;
+ }
+ }
+ }
+
+ for (i = maxlen; i >= 1 ; i--) {
+ /* printf("Analyzing: [%i]...",i); fflush(stdout); */
+ for (currentl = (lenlist+i); currentl != NULL; currentl = currentl->next) {
+ if (currentl->state == NULL)
+ break;
+ if (currentl->state->mergeable != 1)
+ continue;
+ /* Find states hashing to same value as current */
+ state = currentl->state;
+ hash = state->hashval;
+ for (currenth = hashstates+hash; currenth != NULL; currenth = currenth->next) {
+ /* Merge */
+ if (currenth->state != state && currenth->state->mergeable == 1 && currenth->state->distance == state->distance && lexc_eq_paths(currenth->state,state)) {
+ currenth->state->merge_with = state;
+ for (purgestate = currenth->state; purgestate->lexstate == NULL; purgestate = purgestate->trans->target) {
+ purgestate->mergeable = 2;
+ }
+ }
+ }
+ }
+ }
+
+ /* Go through statelist and remove merged states and free states, trans */
+
+ for (s = statelist, sprev = NULL; s != NULL; s = s->next) {
+ for (t = s->state->trans, tprev = NULL; t != NULL; tprev = t, t = t->next) {
+ t->target = t->target->merge_with;
+ if (tprev != NULL && s->state->mergeable == 2) {
+ xxfree(tprev);
+ } else {
+ if (t->target->lexstate != NULL)
+ t->target->lexstate->targeted = 1;
+ }
+ }
+ if (tprev != NULL && s->state->mergeable == 2)
+ xxfree(tprev);
+ }
+ for (s = statelist, sprev = NULL; s != NULL; ) {
+ if (s->state->mergeable == 2) {
+ if (sprev != NULL) {
+ sprev->next = s->next;
+ } else {
+ statelist = s;
+ }
+ xxfree(s->state);
+ sf = s;
+ s = s->next;
+ xxfree(sf);
+ } else {
+ sprev = s;
+ s = s ->next;
+ }
+ }
+
+ /* Cleanup */
+
+ for (i = 0; i < maxlen ; i++) {
+ newl = NULL;
+ for (currentl = (lenlist+i)->next; currentl != NULL ;currentl=currentl->next) {
+ if (newl != NULL)
+ xxfree(newl);
+ newl = currentl;
+ }
+ if (newl != NULL)
+ xxfree(newl);
+ }
+ for (i = 0; i < tablesize ; i++) {
+ newh = NULL;
+ for (currenth = (hashstates+i)->next; currenth != NULL ;currenth=currenth->next) {
+ if (newh != NULL)
+ xxfree(newh);
+ newh = currenth;
+ }
+ if (newh != NULL)
+ xxfree(newh);
+ }
+ xxfree(hashstates);
+ xxfree(lenlist);
+}
+
+struct fsm *lexc_to_fsm() {
+ struct statelist *s, *sa;
+ struct fsm_state *fsm;
+ struct fsm *net;
+ struct trans *t;
+ int i, j, linecount;
+
+ fprintf(stderr,"Building lexicon...\n");
+ fflush(stdout);
+ lexc_merge_states();
+ net = fsm_create("");
+ xxfree(net->sigma);
+ net->sigma = lexsigma;
+ lexc_number_states();
+ if (hasfinal == 0) {
+ fprintf(stderr,"Warning: # is never reached!!!\n");
+ return(fsm_empty_set());
+ }
+ sa = xxmalloc(sizeof(struct statelist)*lexc_statecount);
+ for (s = statelist; s != NULL; s = s->next) {
+ sa[s->state->number].state = s->state;
+ sa[s->state->number].start = s->start;
+ sa[s->state->number].final = s->final;
+ }
+ linecount = 0;
+ for (s = statelist; s != NULL; s = s->next) {
+ linecount++;
+ for (t = s->state->trans; t != NULL; t = t->next)
+ linecount++;
+ }
+ fsm = xxmalloc(sizeof(struct fsm_state)*(linecount+1));
+ for (i = 0, j = 0, s = sa; j < lexc_statecount; j++) {
+ if (s[j].state->trans == NULL) {
+ add_fsm_arc(fsm,i,s[j].state->number, -1, -1, -1, s[j].final, s[j].start);
+ i++;
+ } else {
+ for (t = s[j].state->trans; t != NULL; t = t->next) {
+ add_fsm_arc(fsm,i,s[j].state->number,t->in,t->out,t->target->number,s[j].final,s[j].start);
+ i++;
+ }
+ }
+ }
+ add_fsm_arc(fsm, i, -1, -1, -1, -1, -1, -1);
+ net->states = fsm;
+ net->statecount = lexc_statecount;
+ fsm_update_flags(net, UNK, UNK, UNK, UNK, UNK, UNK);
+ if (sigma_find_number(EPSILON, lexsigma) == -1)
+ sigma_add_special(EPSILON, lexsigma);
+ xxfree(s);
+ lexc_cleanup();
+ sigma_cleanup(net,0);
+ sigma_sort(net);
+
+ fprintf(stderr,"Determinizing...\n");
+ fflush(stdout);
+ net = fsm_determinize(net);
+ fprintf(stderr,"Minimizing...\n");
+ fflush(stdout);
+ net = fsm_topsort(fsm_minimize(net));
+ fprintf(stderr,"Done!\n");
+ return(net);
+}
+
+void lexc_cleanup() {
+ struct lexstates *l, *ln;
+ struct statelist *s, *sn;
+ struct trans *t, *tn;
+ struct multichar_symbols *mcs, *mcsn;
+ struct lexc_hashtable *lhash, *lprev;
+ int i;
+ xxfree(mchash);
+ for (i=0; i < SIGMA_HASH_TABLESIZE; i++) {
+ for (lhash = hashtable+i; lhash != NULL; ) {
+ if (lhash->symbol != NULL) {
+ xxfree(lhash->symbol);
+ }
+ lprev = lhash;
+ lhash = lhash->next;
+ if (lprev != hashtable+i) { xxfree(lprev); }
+ }
+ }
+ xxfree(hashtable);
+ for (mcs = mc ; mcs != NULL ; mcs = mcsn) {
+ mcsn = mcs->next;
+ xxfree(mcs->symbol);
+ xxfree(mcs);
+ }
+ for (l = lexstates ; l != NULL ; l = ln) {
+ ln = l->next;
+ xxfree(l->name);
+ xxfree(l);
+ }
+ for (s = statelist; s != NULL; s = s->next) {
+ for (t = s->state->trans; t != NULL; t = tn) {
+ tn = t->next;
+ xxfree(t);
+ }
+ xxfree(s->state);
+ }
+ for (s = statelist; s != NULL; s = sn) {
+ sn = s->next;
+ xxfree(s);
+ }
+}
diff --git a/mem.c b/mem.c
new file mode 100644
index 0000000..a861ba1
--- /dev/null
+++ b/mem.c
@@ -0,0 +1,95 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include "foma.h"
+#include <stdlib.h>
+#include <string.h>
+
+/* Global variables */
+int g_show_flags = 0;
+int g_obey_flags = 1;
+int g_flag_is_epsilon = 0;
+int g_print_space = 0;
+int g_print_pairs = 0;
+int g_minimal = 1;
+int g_name_nets = 0;
+int g_print_sigma = 1;
+int g_quit_on_fail = 1;
+int g_quote_special = 0;
+int g_recursive_define = 0;
+int g_sort_arcs = 1;
+int g_verbose = 1;
+int g_minimize_hopcroft = 1;
+int g_compose_tristate = 0;
+int g_list_limit = 100;
+int g_list_random_limit = 15;
+int g_med_limit = 3;
+int g_med_cutoff = 15;
+int g_lexc_align = 0;
+char *g_att_epsilon = "@0@";
+
+char *xxstrndup(const char *s, size_t n) {
+ char *r = NULL;
+ const char *p = s;
+ while(*p++ && n--);
+ n = p - s - 1;
+ r = (char *) xxmalloc(n + 1);
+ if(r != NULL) {
+ memcpy(r, s, n);
+ r[n] = 0;
+ }
+ return r;
+}
+
+int next_power_of_two(int v) {
+ int i;
+ for (i=0; v > 0; i++)
+ v = v >> 1;
+ return (1 << i);
+}
+
+unsigned int round_up_to_power_of_two(unsigned int v) {
+ v--;
+ v |= v >> 1;
+ v |= v >> 2;
+ v |= v >> 4;
+ v |= v >> 8;
+ v |= v >> 16;
+ v++;
+ return(v);
+}
+
+INLINE void *xxmalloc(size_t size) {
+ return(malloc(size));
+}
+
+INLINE void xxfree(void *ptr) {
+ free(ptr);
+}
+
+void *xxrealloc(void *ptr, size_t size) {
+ return(realloc(ptr, size));
+}
+
+INLINE void *xxcalloc(size_t nmemb, size_t size) {
+ return(calloc(nmemb,size));
+}
+
+INLINE char *xxstrdup(const char *s) {
+ return(strdup(s));
+}
+
diff --git a/minimize.c b/minimize.c
new file mode 100644
index 0000000..303e11b
--- /dev/null
+++ b/minimize.c
@@ -0,0 +1,673 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include <assert.h>
+#include <limits.h>
+#include <stdint.h>
+#include "foma.h"
+
+static struct fsm *fsm_minimize_brz(struct fsm *net);
+static struct fsm *fsm_minimize_hop(struct fsm *net);
+static struct fsm *rebuild_machine(struct fsm *net);
+
+static int *single_sigma_array, *double_sigma_array, *memo_table, *temp_move, *temp_group, maxsigma, epsilon_symbol, num_states, num_symbols, num_finals, mainloop, total_states;
+
+static _Bool *finals;
+
+struct statesym {
+ int target;
+ unsigned short int symbol;
+ struct state_list *states;
+ struct statesym *next;
+};
+
+struct state_list {
+ int state;
+ struct state_list *next;
+};
+
+struct p {
+ struct e *first_e;
+ struct e *last_e;
+ struct p *current_split;
+ struct p *next;
+ struct agenda *agenda;
+ int count;
+ int t_count;
+ int inv_count;
+ int inv_t_count;
+};
+
+struct e {
+ struct p *group;
+ struct e *left;
+ struct e *right;
+ int inv_count;
+};
+
+struct agenda {
+ struct p *p;
+ struct agenda *next;
+ _Bool index;
+};
+
+struct trans_list {
+ int inout;
+ int source;
+} *trans_list;
+
+struct trans_array {
+ struct trans_list *transitions;
+ unsigned int size;
+ unsigned int tail;
+} *trans_array;
+
+
+
+static struct p *P, *Phead, *Pnext, *current_w;
+static struct e *E;
+static struct agenda *Agenda_head, *Agenda_top, *Agenda_next, *Agenda;
+
+static INLINE int refine_states(int sym);
+static void init_PE();
+static void agenda_add(struct p *pptr, int start);
+static void sigma_to_pairs(struct fsm *net);
+/* static void single_symbol_to_symbol_pair(int symbol, int *symbol_in, int *symbol_out); */
+static INLINE int symbol_pair_to_single_symbol(int in, int out);
+static void generate_inverse(struct fsm *net);
+
+struct fsm *fsm_minimize(struct fsm *net) {
+ extern int g_minimal;
+ extern int g_minimize_hopcroft;
+
+ if (net == NULL) { return NULL; }
+ /* The network needs to be deterministic and trim before we minimize */
+ if (net->is_deterministic != YES)
+ net = fsm_determinize(net);
+ if (net->is_pruned != YES)
+ net = fsm_coaccessible(net);
+ if (net->is_minimized != YES && g_minimal == 1) {
+ if (g_minimize_hopcroft != 0) {
+ net = fsm_minimize_hop(net);
+ }
+ else
+ net = fsm_minimize_brz(net);
+ fsm_update_flags(net,YES,YES,YES,YES,UNK,UNK);
+ }
+ return(net);
+}
+
+static struct fsm *fsm_minimize_brz(struct fsm *net) {
+ return(fsm_determinize(fsm_reverse(fsm_determinize(fsm_reverse(net)))));
+}
+
+static struct fsm *fsm_minimize_hop(struct fsm *net) {
+
+ struct e *temp_E;
+ struct trans_array *tptr;
+ struct trans_list *transitions;
+ int i,j,minsym,next_minsym,current_i, stateno, thissize, source;
+ unsigned int tail;
+
+ fsm_count(net);
+ if (net->finalcount == 0) {
+ fsm_destroy(net);
+ return(fsm_empty_set());
+ }
+
+ num_states = net->statecount;
+
+ P = NULL;
+
+ /*
+ 1. generate the inverse lookup table
+ 2. generate P and E (partitions, states linked list)
+ 3. Init Agenda = {Q, Q-F}
+ 4. Split until Agenda is empty
+ */
+
+ sigma_to_pairs(net);
+
+ init_PE();
+
+ if (total_states == num_states) {
+ goto bail;
+ }
+
+ generate_inverse(net);
+
+
+ Agenda_head->index = 0;
+ if (Agenda_head->next != NULL)
+ Agenda_head->next->index = 0;
+
+ for (Agenda = Agenda_head; Agenda != NULL; ) {
+ /* Remove current_w from agenda */
+ current_w = Agenda->p;
+ current_i = Agenda->index;
+ Agenda->p->agenda = NULL;
+ Agenda = Agenda->next;
+
+ /* Store current group state number in tmp_group */
+ /* And figure out minsym */
+ /* If index is 0 we start splitting from the first symbol */
+ /* Otherwise we split from where we left off last time */
+
+ thissize = 0;
+ minsym = INT_MAX;
+ for (temp_E = current_w->first_e; temp_E != NULL; temp_E = temp_E->right) {
+ stateno = temp_E - E;
+ *(temp_group+thissize) = stateno;
+ thissize++;
+ tptr = trans_array+stateno;
+ /* Clear tails if symloop should start from 0 */
+ if (current_i == 0)
+ tptr->tail = 0;
+
+ tail = tptr->tail;
+ transitions = (tptr->transitions)+tail;
+ if (tail < tptr->size && transitions->inout < minsym) {
+ minsym = transitions->inout;
+ }
+ }
+
+ for (next_minsym = INT_MAX; minsym != INT_MAX ; minsym = next_minsym, next_minsym = INT_MAX) {
+
+ /* Add states to temp_move */
+ for (i = 0, j = 0; i < thissize; i++) {
+ tptr = trans_array+*(temp_group+i);
+ tail = tptr->tail;
+ transitions = (tptr->transitions)+tail;
+ while (tail < tptr->size && transitions->inout == minsym) {
+ source = transitions->source;
+ if (*(memo_table+(source)) != mainloop) {
+ *(memo_table+(source)) = mainloop;
+ *(temp_move+j) = source;
+ j++;
+ }
+ tail++;
+ transitions++;
+ }
+ tptr->tail = tail;
+ if (tail < tptr->size && transitions->inout < next_minsym) {
+ next_minsym = transitions->inout;
+ }
+ }
+ if (j == 0) {
+ continue;
+ }
+ mainloop++;
+ if (refine_states(j) == 1) {
+ break; /* break loop if we split current_w */
+ }
+ }
+ if (total_states == num_states) {
+ break;
+ }
+ }
+
+ net = rebuild_machine(net);
+
+ xxfree(trans_array);
+ xxfree(trans_list);
+
+ bail:
+
+ xxfree(Agenda_top);
+
+ xxfree(memo_table);
+ xxfree(temp_move);
+ xxfree(temp_group);
+
+
+ xxfree(finals);
+ xxfree(E);
+ xxfree(Phead);
+ xxfree(single_sigma_array);
+ xxfree(double_sigma_array);
+
+ return(net);
+}
+
+static struct fsm *rebuild_machine(struct fsm *net) {
+ int i,j, group_num, source, target, new_linecount = 0, arccount = 0;
+ struct fsm_state *fsm;
+ struct p *myp;
+ struct e *thise;
+
+ if (net->statecount == total_states) {
+ return(net);
+ }
+ fsm = net->states;
+
+ /* We need to make sure state 0 is first in its group */
+ /* to get the proper numbering of states */
+
+ if (E->group->first_e != E) {
+ E->group->first_e = E;
+ }
+
+ /* Recycling t_count for group numbering use here */
+
+ group_num = 1;
+ myp = P;
+ while (myp != NULL) {
+ myp->count = 0;
+ myp = myp->next;
+ }
+
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ thise = E+((fsm+i)->state_no);
+ if (thise->group->first_e == thise) {
+ new_linecount++;
+ if ((fsm+i)->start_state == 1) {
+ thise->group->t_count = 0;
+ thise->group->count = 1;
+ } else if (thise->group->count == 0) {
+ thise->group->t_count = group_num++;
+ thise->group->count = 1;
+ }
+ }
+ }
+
+ for (i=0, j=0; (fsm+i)->state_no != -1; i++) {
+ thise = E+((fsm+i)->state_no);
+ if (thise->group->first_e == thise) {
+ source = thise->group->t_count;
+ target = ((fsm+i)->target == -1) ? -1 : (E+((fsm+i)->target))->group->t_count;
+ add_fsm_arc(fsm, j, source, (fsm+i)->in, (fsm+i)->out, target, finals[(fsm+i)->state_no], (fsm+i)->start_state);
+ arccount = ((fsm+i)->target == -1) ? arccount : arccount+1;
+ j++;
+ }
+ }
+
+ add_fsm_arc(fsm, j, -1, -1, -1, -1, -1, -1);
+ fsm = xxrealloc(fsm,sizeof(struct fsm_state)*(new_linecount+1));
+ net->states = fsm;
+ net->linecount = j+1;
+ net->arccount = arccount;
+ net->statecount = total_states;
+ return(net);
+}
+
+static INLINE int refine_states(int invstates) {
+ int i, selfsplit;
+ struct e *thise;
+ struct p *tP, *newP = NULL;
+
+ /*
+ 1. add inverse(P,a) to table of inverses, disallowing duplicates
+ 2. first pass on S, touch each state once, increasing P->t_count
+ 3. for each P where counter != count, split and add to agenda
+ */
+
+ /* Inverse to table of inverses */
+ selfsplit = 0;
+
+ /* touch and increase P->counter */
+ for (i=0; i < invstates; i++) {
+ ((E+(*(temp_move+i)))->group)->t_count++;
+ ((E+(*(temp_move+i)))->group)->inv_t_count += ((E+(*(temp_move+i)))->inv_count);
+ assert((E+(*(temp_move+i)))->group->t_count <= (E+(*(temp_move+i)))->group->count);
+ }
+
+ /* Split (this is the tricky part) */
+
+ for (i=0; i < invstates; i++) {
+
+ thise = E+*(temp_move+i);
+ tP = thise->group;
+
+ /* Do we need to split?
+ if we've touched as many states as there are in the partition
+ we don't split */
+
+ if (tP->t_count == tP->count) {
+ tP->t_count = 0;
+ tP->inv_t_count = 0;
+ continue;
+ }
+
+ if ((tP->t_count != tP->count) && (tP->count > 1) && (tP->t_count > 0)) {
+
+ /* Check if we already split this */
+ newP = tP->current_split;
+ if (newP == NULL) {
+ /* printf("tP [%i] newP [%i]\n",tP->inv_count,tP->inv_t_count); */
+ /* Create new group newP */
+ total_states++;
+ if (total_states == num_states)
+ return(1); /* Abort now, machine is already minimal */
+ tP->current_split = Pnext++;
+ newP = tP->current_split;
+ newP->first_e = newP->last_e = thise;
+ newP->count = 0;
+ newP->inv_count = tP->inv_t_count;
+ newP->inv_t_count = 0;
+ newP->t_count = 0;
+ newP->current_split = NULL;
+ newP->agenda = NULL;
+
+ /* Add to agenda */
+
+ /* If the current block (tP) is on the agenda, we add both back */
+ /* to the agenda */
+ /* In practice we need only add newP since tP stays where it is */
+ /* However, we mark the larger one as not starting the symloop */
+ /* from zero */
+ if (tP->agenda != NULL) {
+ /* Is tP smaller */
+ if (tP->inv_count < tP->inv_t_count) {
+ agenda_add(newP, 1);
+ tP->agenda->index = 0;
+ }
+ else {
+ agenda_add(newP, 0);
+ }
+ /* In the event that we're splitting the partition we're currently */
+ /* splitting with, we can simply add both new partitions to the agenda */
+ /* and break out of the entire sym loop after we're */
+ /* done with the current sym and move on with the agenda */
+ /* We process the larger one for all symbols */
+ /* and the smaller one for only the ones remaining in this symloop */
+
+ } else if (tP == current_w) {
+ agenda_add(((tP->inv_count < tP->inv_t_count) ? tP : newP),0);
+ agenda_add(((tP->inv_count >= tP->inv_t_count) ? tP : newP),1);
+ selfsplit = 1;
+ } else {
+ /* If the block is not on the agenda, we add */
+ /* the smaller of tP, newP and start the symloop from 0 */
+ agenda_add((tP->inv_count < tP->inv_t_count ? tP : newP),0);
+ }
+ /* Add to middle of P-chain */
+ newP->next = P->next;
+ P->next = newP;
+ }
+
+ thise->group = newP;
+ newP->count++;
+
+ /* need to make tP->last_e point to the last untouched e */
+ if (thise == tP->last_e)
+ tP->last_e = thise->left;
+ if (thise == tP->first_e)
+ tP->first_e = thise->right;
+
+ /* Adjust links */
+ if (thise->left != NULL)
+ thise->left->right = thise->right;
+ if (thise->right != NULL)
+ thise->right->left = thise->left;
+
+ if (newP->last_e != thise) {
+ newP->last_e->right = thise;
+ thise->left = newP->last_e;
+ newP->last_e = thise;
+ }
+
+ thise->right = NULL;
+ if (newP->first_e == thise)
+ thise->left = NULL;
+
+ /* Are we done for this block? Adjust counters */
+ if (newP->count == tP->t_count) {
+ tP->count = tP->count - newP->count;
+ tP->inv_count = tP->inv_count - tP->inv_t_count;
+ tP->current_split = NULL;
+ tP->t_count = 0;
+ tP->inv_t_count = 0;
+ }
+ }
+ }
+ /* We return 1 if we just split the partition we were working with */
+ return (selfsplit);
+}
+
+static void agenda_add(struct p *pptr, int start) {
+
+ /* Use FILO strategy here */
+
+ struct agenda *ag;
+ //ag = xxmalloc(sizeof(struct agenda));
+ ag = Agenda_next++;
+ if (Agenda != NULL)
+ ag->next = Agenda;
+ else
+ ag->next = NULL;
+ ag->p = pptr;
+ ag->index = start;
+ Agenda = ag;
+ pptr->agenda = ag;
+}
+
+static void init_PE() {
+ /* Create two members of P
+ (nonfinals,finals)
+ and put both of them on the agenda
+ */
+
+ int i;
+ struct e *last_f, *last_nonf;
+ struct p *nonFP, *FP;
+ struct agenda *ag;
+
+ mainloop = 1;
+ memo_table = xxcalloc(num_states,sizeof(int));
+ temp_move = xxcalloc(num_states,sizeof(int));
+ temp_group = xxcalloc(num_states,sizeof(int));
+ Phead = P = Pnext = xxcalloc(num_states+1, sizeof(struct p));
+ nonFP = Pnext++;
+ FP = Pnext++;
+ nonFP->next = FP;
+ nonFP->count = num_states-num_finals;
+ FP->next = NULL;
+ FP->count = num_finals;
+ FP->t_count = 0;
+ nonFP->t_count = 0;
+ FP->current_split = NULL;
+ nonFP->current_split = NULL;
+ FP->inv_count = nonFP->inv_count = FP->inv_t_count = nonFP->inv_t_count = 0;
+
+ /* How many groups can we put on the agenda? */
+ Agenda_top = Agenda_next = xxcalloc(num_states*2, sizeof(struct agenda));
+ Agenda_head = NULL;
+
+ P = NULL;
+ total_states = 0;
+
+ if (num_finals > 0) {
+ ag = Agenda_next++;
+ FP->agenda = ag;
+ P = FP;
+ P->next = NULL;
+ ag->p = FP;
+ Agenda_head = ag;
+ ag->next = NULL;
+ total_states++;
+ }
+ if (num_states - num_finals > 0) {
+ ag = Agenda_next++;
+ nonFP->agenda = ag;
+ ag->p = nonFP;
+ ag->next = NULL;
+ total_states++;
+ if (Agenda_head != NULL) {
+ Agenda_head->next = ag;
+ P->next = nonFP;
+ P->next->next = NULL;
+ } else {
+ P = nonFP;
+ P->next = NULL;
+ Agenda_head = ag;
+ }
+ }
+
+ /* Initialize doubly linked list E */
+ E = xxcalloc(num_states,sizeof(struct e));
+
+ last_f = NULL;
+ last_nonf = NULL;
+
+ for (i=0; i < num_states; i++) {
+ if (finals[i]) {
+ (E+i)->group = FP;
+ (E+i)->left = last_f;
+ if (i > 0 && last_f != NULL)
+ last_f->right = (E+i);
+ if (last_f == NULL)
+ FP->first_e = (E+i);
+ last_f = (E+i);
+ FP->last_e = (E+i);
+ } else {
+ (E+i)->group = nonFP;
+ (E+i)->left = last_nonf;
+ if (i > 0 && last_nonf != NULL)
+ last_nonf->right = (E+i);
+ if (last_nonf == NULL)
+ nonFP->first_e = (E+i);
+ last_nonf = (E+i);
+ nonFP->last_e = (E+i);
+ }
+ (E+i)->inv_count = 0;
+ }
+
+ if (last_f != NULL)
+ last_f->right = NULL;
+ if (last_nonf != NULL)
+ last_nonf->right = NULL;
+}
+
+static int trans_sort_cmp(const void *a, const void *b) {
+ return (((const struct trans_list *)a)->inout - ((const struct trans_list *)b)->inout);
+}
+
+static void generate_inverse(struct fsm *net) {
+
+ struct fsm_state *fsm;
+ struct trans_array *tptr;
+ struct trans_list *listptr;
+
+ int i, source, target, offsetcount, symbol, size;
+ fsm = net->states;
+ trans_array = xxcalloc(net->statecount, sizeof(struct trans_array));
+ trans_list = xxcalloc(net->arccount, sizeof(struct trans_list));
+
+ /* Figure out the number of transitions each one has */
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target == -1) {
+ continue;
+ }
+ target = (fsm+i)->target;
+ (E+target)->inv_count++;
+ (E+target)->group->inv_count++;
+ (trans_array+target)->size++;
+ }
+ offsetcount = 0;
+ for (i=0; i < net->statecount; i++) {
+ (trans_array+i)->transitions = trans_list + offsetcount;
+ offsetcount += (trans_array+i)->size;
+ }
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target == -1) {
+ continue;
+ }
+ symbol = symbol_pair_to_single_symbol((fsm+i)->in,(fsm+i)->out);
+ source = (fsm+i)->state_no;
+ target = (fsm+i)->target;
+ tptr = trans_array + target;
+ ((tptr->transitions)+(tptr->tail))->inout = symbol;
+ ((tptr->transitions)+(tptr->tail))->source = source;
+ tptr->tail++;
+ }
+ /* Sort arcs */
+ for (i=0; i < net->statecount; i++) {
+ listptr = (trans_array+i)->transitions;
+ size = (trans_array+i)->size;
+ if (size > 1)
+ qsort(listptr, size, sizeof(struct trans_list), trans_sort_cmp);
+ }
+}
+
+static void sigma_to_pairs(struct fsm *net) {
+
+ int i, j, x, y, z, next_x = 0;
+ struct fsm_state *fsm;
+
+ fsm = net->states;
+
+ epsilon_symbol = -1;
+ maxsigma = sigma_max(net->sigma);
+
+ maxsigma++;
+
+ single_sigma_array = xxmalloc(2*maxsigma*maxsigma*sizeof(int));
+ double_sigma_array = xxmalloc(maxsigma*maxsigma*sizeof(int));
+
+ for (i=0; i < maxsigma; i++) {
+ for (j=0; j< maxsigma; j++) {
+ *(double_sigma_array+maxsigma*i+j) = -1;
+ }
+ }
+
+ /* f(x) -> y,z sigma pair */
+ /* f(y,z) -> x simple entry */
+ /* if exists f(n) <-> EPSILON, EPSILON, save n */
+ /* symbol(x) x>=1 */
+
+ /* Forward mapping: */
+ /* *(double_sigma_array+maxsigma*in+out) */
+
+ /* Backmapping: */
+ /* *(single_sigma_array+(symbol*2) = in(symbol) */
+ /* *(single_sigma_array+(symbol*2+1) = out(symbol) */
+
+ /* Table for checking whether a state is final */
+
+ finals = xxcalloc(num_states, sizeof(_Bool));
+ x = 0; num_finals = 0;
+ net->arity = 1;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->final_state == 1 && finals[(fsm+i)->state_no] != 1) {
+ num_finals++;
+ finals[(fsm+i)->state_no] = 1;
+ }
+ y = (fsm+i)->in;
+ z = (fsm+i)->out;
+ if (y != z || y == UNKNOWN || z == UNKNOWN)
+ net->arity = 2;
+ if ((y == -1) || (z == -1))
+ continue;
+ if (*(double_sigma_array+maxsigma*y+z) == -1) {
+ *(double_sigma_array+maxsigma*y+z) = x;
+ *(single_sigma_array+next_x) = y;
+ next_x++;
+ *(single_sigma_array+next_x) = z;
+ next_x++;
+ if (y == EPSILON && z == EPSILON) {
+ epsilon_symbol = x;
+ }
+ x++;
+ }
+ }
+ num_symbols = x;
+}
+
+static INLINE int symbol_pair_to_single_symbol(int in, int out) {
+ return(*(double_sigma_array+maxsigma*in+out));
+}
diff --git a/python/foma.py b/python/foma.py
new file mode 100644
index 0000000..4dad295
--- /dev/null
+++ b/python/foma.py
@@ -0,0 +1,495 @@
+# -*- coding: utf-8 -*-
+
+# Foma: a finite-state toolkit and library. #
+# Copyright © 2008-2015 Mans Hulden #
+
+# This file is part of foma. #
+
+# Licensed under the Apache License, Version 2.0 (the "License"); #
+# you may not use this file except in compliance with the License. #
+# You may obtain a copy of the License at #
+
+# http://www.apache.org/licenses/LICENSE-2.0 #
+
+# Unless required by applicable law or agreed to in writing, software #
+# distributed under the License is distributed on an "AS IS" BASIS, #
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #
+# See the License for the specific language governing permissions and #
+# limitations under the License. #
+
+from sys import maxint
+from ctypes import *
+from ctypes.util import find_library
+
+fomalibpath = find_library('foma')
+foma = cdll.LoadLibrary(fomalibpath)
+
+class FSTstruct(Structure):
+
+ _fields_ = [("name", c_char * 40), ("arity", c_int), ("arccount", c_int), ("statecount", c_int), ("linecount", c_int), ("finalcount", c_int), ("pathcount", c_longlong), ("is_deterministic", c_int), ("is_pruned", c_int), ("is_minimized", c_int), ("is_epsilon_free", c_int), ("is_loop_free", c_int), ("is_completed", c_int), ("arcs_sorted_in", c_int), ("arcs_sorted_out", c_int), ("fsm_state", c_void_p), ("sigma", c_void_p), ("medlookup", c_void_p)]
+
+foma_fsm_parse_regex = foma.fsm_parse_regex
+foma_fsm_parse_regex.restype = POINTER(FSTstruct)
+foma_apply_init = foma.apply_init
+foma_apply_init.restype = c_void_p
+foma_apply_clear = foma.apply_clear
+foma_apply_words = foma.apply_words
+foma_apply_words.restype = c_char_p
+foma_apply_lower_words = foma.apply_lower_words
+foma_apply_lower_words.restype = c_char_p
+foma_apply_upper_words = foma.apply_upper_words
+foma_apply_upper_words.restype = c_char_p
+foma_apply_down = foma.apply_down
+foma_apply_down.restype = c_char_p
+foma_apply_up = foma.apply_up
+foma_apply_up.restype = c_char_p
+foma_apply_set_space_symbol = foma.apply_set_space_symbol
+foma_fsm_count = foma.fsm_count
+foma_fsm_topsort = foma.fsm_topsort
+foma_fsm_topsort.restype = POINTER(FSTstruct)
+foma_fsm_minimize = foma.fsm_minimize
+foma_fsm_minimize.restype = POINTER(FSTstruct)
+foma_fsm_union = foma.fsm_union
+foma_fsm_union.restype = POINTER(FSTstruct)
+foma_fsm_intersect = foma.fsm_intersect
+foma_fsm_intersect.restype = POINTER(FSTstruct)
+foma_fsm_minus = foma.fsm_minus
+foma_fsm_minus.restype = POINTER(FSTstruct)
+foma_fsm_compose = foma.fsm_compose
+foma_fsm_compose.restype = POINTER(FSTstruct)
+foma_fsm_concat = foma.fsm_concat
+foma_fsm_concat.restype = POINTER(FSTstruct)
+foma_fsm_copy = foma.fsm_copy
+foma_fsm_copy.restype = POINTER(FSTstruct)
+foma_fsm_complement = foma.fsm_complement
+foma_fsm_complement.restype = POINTER(FSTstruct)
+foma_fsm_lower = foma.fsm_lower
+foma_fsm_lower.restype = POINTER(FSTstruct)
+foma_fsm_upper = foma.fsm_upper
+foma_fsm_upper.restype = POINTER(FSTstruct)
+foma_fsm_minimize = foma.fsm_minimize
+foma_fsm_minimize.restype = POINTER(FSTstruct)
+foma_fsm_destroy = foma.fsm_destroy
+foma_fsm_equivalent = foma.fsm_equivalent
+foma_fsm_equivalent.restype = c_int
+foma_fsm_isempty = foma.fsm_isempty
+foma_fsm_isempty.restype = c_int
+foma_fsm_flatten = foma.fsm_flatten
+foma_fsm_flatten.restype = POINTER(FSTstruct)
+foma_apply_set_space_symbol = foma.apply_set_space_symbol
+foma_fsm_read_binary_file = foma.fsm_read_binary_file
+foma_fsm_read_binary_file.restype = POINTER(FSTstruct)
+
+
+"""Define functions."""
+foma_add_defined = foma.add_defined
+foma_add_defined.restype = c_int
+foma_add_defined_function = foma.add_defined_function
+foma_add_defined_function.restype = c_int
+defined_networks_init = foma.defined_networks_init
+defined_networks_init.restype = c_void_p
+defined_functions_init = foma.defined_functions_init
+defined_functions_init.restype = c_void_p
+
+"""Trie functions."""
+fsm_trie_init = foma.fsm_trie_init
+fsm_trie_init.restype = c_void_p
+fsm_trie_add_word = foma.fsm_trie_add_word
+fsm_trie_done = foma.fsm_trie_done
+fsm_trie_done.restype = POINTER(FSTstruct)
+
+class FSTnetworkdefinitions(object):
+
+ def __init__(self):
+ self.defhandle = defined_networks_init(None)
+
+class FSTfunctiondefinitions(object):
+
+ def __init__(self):
+ self.deffhandle = defined_functions_init(None)
+
+
+class FST(object):
+
+ networkdefinitions = FSTnetworkdefinitions()
+ functiondefinitions = FSTfunctiondefinitions()
+
+ @classmethod
+ def define(cls, definition, name):
+ """Defines an FSM constant; can be supplied regex or existing FSM."""
+ name = cls.encode(name)
+ if isinstance(definition, FST):
+ retval = foma.add_defined(c_void_p(cls.networkdefinitions.defhandle), foma_fsm_copy(definition.fsthandle), c_char_p(name))
+ elif isinstance(definition, basestring):
+ regex = cls.encode(definition)
+ retval = foma.add_defined(c_void_p(cls.networkdefinitions.defhandle), foma_fsm_parse_regex(c_char_p(regex), c_void_p(cls.networkdefinitions.defhandle), c_void_p(cls.functiondefinitions.deffhandle)), c_char_p(name))
+ else:
+ raise ValueError("Expected str, unicode, or FSM")
+
+ @classmethod
+ def definef(cls, prototype, definition):
+ """Defines an FSM function."""
+ # Prototype is a 2-tuple (name, (arg1name, ..., argname))
+ # Definition is regex using prototype variables
+ name = cls.encode(prototype[0] + '(')
+ if isinstance(definition, basestring):
+ numargs = len(prototype[1])
+ for i in xrange(numargs):
+ definition = definition.replace(prototype[1][i], "@ARGUMENT0%i@" % (i+1))
+ regex = cls.encode(definition + ';')
+ retval = foma.add_defined_function(c_void_p(cls.functiondefinitions.deffhandle), c_char_p(name), c_char_p(regex), c_int(numargs))
+ else:
+ raise ValueError("Expected regex as definition")
+
+ @classmethod
+ def wordlist(cls, wordlist, minimize = True):
+ """Create FSM directly from wordlist.
+ Returns a trie-shaped deterministic automaton if not minimized."""
+ th = fsm_trie_init()
+ for w in wordlist:
+ thisword = cls.encode(w)
+ fsm_trie_add_word(c_void_p(th), c_char_p(thisword))
+ fsm = cls()
+ fsm.fsthandle = fsm_trie_done(c_void_p(th))
+ if minimize:
+ fsm.fsthandle = foma_fsm_minimize(fsm.fsthandle)
+ return fsm
+
+ @classmethod
+ def load(cls, filename):
+ """Load binary FSM from file."""
+ fsm = cls()
+ fsm.fsthandle = foma_fsm_read_binary_file(c_char_p(filename))
+ if not fsm.fsthandle:
+ raise ValueError("File error.")
+ return fsm
+
+ @staticmethod
+ def encode(string):
+ """Makes sure str and unicode are converted."""
+ if isinstance(string, unicode):
+ return string.encode('utf8')
+ elif isinstance(string, str):
+ return string
+ else:
+ return str(string)
+
+ def __init__(self, regex = False):
+ if regex:
+ self.regex = self.encode(regex)
+ self.fsthandle = foma_fsm_parse_regex(c_char_p(self.regex), c_void_p(self.networkdefinitions.defhandle), c_void_p(self.functiondefinitions.deffhandle))
+ if not self.fsthandle:
+ raise ValueError("Syntax error in regex")
+ else:
+ self.fsthandle = None
+ self.getitemapplyer = None
+
+ def __getitem__(self, key):
+ if not self.fsthandle:
+ raise KeyError('FST not defined')
+ if not self.getitemapplyer:
+ self.getitemapplyer = foma_apply_init(self.fsthandle)
+ result = []
+ output = foma_apply_down(c_void_p(self.getitemapplyer), c_char_p(self.encode(key)))
+ while True:
+ if output == None:
+ return result
+ else:
+ result.append(output)
+ output = foma_apply_down(c_void_p(self.getitemapplyer), None)
+
+ def __del__(self):
+ if self.fsthandle:
+ foma_fsm_destroy(self.fsthandle)
+
+ def __str__(self):
+ if not self.fsthandle:
+ raise ValueError('FSM not defined')
+ foma_fsm_count(self.fsthandle)
+ s = 'Name: %s\n' % self.fsthandle.contents.name
+ s += 'States: %i\n' % self.fsthandle.contents.statecount
+ s += 'Transitions: %i\n' % self.fsthandle.contents.arccount
+ s += 'Final states: %i\n' % self.fsthandle.contents.finalcount
+ s += 'Deterministic: %i\n' % self.fsthandle.contents.is_deterministic
+ s += 'Minimized: %i\n' % self.fsthandle.contents.is_minimized
+ s += 'Arity: %i\n' % self.fsthandle.contents.arity
+ return s
+
+ def __len__(self):
+ if self.fsthandle:
+ if self.fsthandle.contents.pathcount == -3: # UNKNOWN
+ self.fsthandle = foma_fsm_topsort(self.fsthandle)
+ if self.fsthandle.contents.pathcount == -1: # CYCLIC
+ raise ValueError("FSM is cyclic")
+ if self.fsthandle.contents.pathcount == -2: # OVERFLOW
+ return maxint
+ return self.fsthandle.contents.pathcount
+ else:
+ raise ValueError("FSM not defined")
+
+ def __add__(self, other):
+ return self.concat(other)
+
+ def __sub__(self, other):
+ return self.minus(other)
+
+ def __le__(self, other):
+ if self.fsthandle and other.fsthandle:
+ return bool(c_int(foma_fsm_isempty(foma_fsm_minimize(foma_fsm_minus(foma_fsm_copy(self.fsthandle),foma_fsm_copy(other.fsthandle))))))
+ else:
+ raise ValueError('Undefined FST')
+
+ def __lt__(self, other):
+ if self.fsthandle and other.fsthandle:
+ return (not self.__eq__(other)) and bool(c_int(foma_fsm_isempty(foma_fsm_minimize(foma_fsm_minus(foma_fsm_copy(self.fsthandle),foma_fsm_copy(other.fsthandle))))))
+ else:
+ raise ValueError('Undefined FST')
+
+ def __or__(self, other):
+ return self.union(other)
+
+ def __and__(self, other):
+ return self.intersect(other)
+
+ def __eq__(self, other):
+ if self.fsthandle and other.fsthandle:
+ return bool(c_int(foma_fsm_equivalent(foma_fsm_copy(self.fsthandle), foma_fsm_copy(other.fsthandle))))
+ else:
+ raise ValueError('Undefined FST')
+
+ def __ne__(self, other):
+ return not(self.__eq__(other))
+
+ def __contains__(self, word):
+ af = self.apply_down(word)
+ try:
+ i = af.next()
+ return True
+ except StopIteration:
+ return False
+
+ def __call__(self, other):
+ if isinstance(other, basestring):
+ return FST("{" + other + "}").compose(self)
+ else:
+ return other.compose(self)
+
+ def __invert__(self):
+ new = FST()
+ new.fsthandle = self._fomacallunary(foma_fsm_complement)
+ return new
+
+ def __iter__(self):
+ return self._apply(foma_apply_upper_words, word = None, tokenize = False)
+
+ def _apply(self, applyf, word = None, tokenize = False):
+ if not self.fsthandle:
+ raise ValueError('FST not defined')
+ applyerhandle = foma_apply_init(self.fsthandle)
+ if tokenize:
+ toksym = '\x07'
+ foma_apply_set_space_symbol(c_void_p(applyerhandle), c_char_p(toksym))
+ if word:
+ output = applyf(c_void_p(applyerhandle), c_char_p(self.encode(word)))
+ else:
+ output = applyf(c_void_p(applyerhandle))
+ while True:
+ if output == None:
+ foma_apply_clear(c_void_p(applyerhandle))
+ return
+ else:
+ if tokenize:
+ yield output[:-1].split('\x07')
+ else:
+ yield output
+ if word:
+ output = applyf(c_void_p(applyerhandle), None)
+ else:
+ output = applyf(c_void_p(applyerhandle))
+
+ def words(self, tokenize = False):
+ return self._apply(foma_apply_words, word = None, tokenize = tokenize)
+
+ def lowerwords(self, tokenize = False):
+ return self._apply(foma_apply_lower_words, word = None, tokenize = tokenize)
+
+ def upperwords(self, tokenize = False):
+ return self._apply(foma_apply_upper_words, word = None, tokenize = tokenize)
+
+ def apply_down(self, word, tokenize = False):
+ return self._apply(foma_apply_down, word = word, tokenize = tokenize)
+
+ def apply_up(self, word, tokenize = False):
+ if self.fsthandle:
+ return self._apply(foma_apply_up, word = word, tokenize = tokenize)
+ else:
+ raise ValueError('Undefined FST')
+
+ def _fomacallunary(self, func, minimize = True):
+ if self.fsthandle:
+ handle = func(foma_fsm_copy(self.fsthandle))
+ if minimize:
+ handle = foma_fsm_minimize(handle)
+ return handle
+ else:
+ raise ValueError('Undefined FST')
+
+ def _fomacallbinary(self, other, func, minimize = True):
+ if self.fsthandle and other.fsthandle:
+ handle = func(foma_fsm_copy(self.fsthandle), foma_fsm_copy(other.fsthandle))
+ if minimize:
+ handle = foma_fsm_minimize(handle)
+ return handle
+ else:
+ raise ValueError('Undefined FST')
+
+ def union(self, other, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallbinary(other, foma_fsm_union, minimize)
+ return new
+
+ def intersect(self, other, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallbinary(other, foma_fsm_intersect, minimize)
+ return new
+
+ def minus(self, other, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallbinary(other, foma_fsm_minus, minimize)
+ return new
+
+ def concat(self, other, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallbinary(other, foma_fsm_concat, minimize)
+ return new
+
+ def compose(self, other, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallbinary(other, foma_fsm_compose, minimize)
+ return new
+
+ def lower(self, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallunary(foma_fsm_lower, minimize)
+ return new
+
+ def upper(self, minimize = True):
+ new = FST()
+ new.fsthandle = self._fomacallunary(foma_fsm_upper, minimize)
+ return new
+
+ def flatten(self):
+ new = FST()
+ eps_sym = FST('□')
+ new.fsthandle = foma_fsm_flatten(foma_fsm_copy(self.fsthandle), foma_fsm_copy(eps_sym.fsthandle))
+ return new
+
+class MTFSM(FST):
+
+ def __init__(self, regex = False, numtapes = 2):
+ if isinstance(regex, str) or isinstance(regex, unicode):
+ FST.__init__(self, regex)
+ eps_sym = FST('□')
+ self.fsthandle = foma_fsm_flatten(foma_fsm_copy(self.fsthandle), foma_fsm_copy(eps_sym.fsthandle))
+ self.numtapes = numtapes
+ elif isinstance(regex, FST):
+ self.fsthandle = foma_fsm_copy(regex.fsthandle)
+ self.regex = None
+ self.numtapes = numtapes
+ else:
+ self.fsthandle = None
+ self.regex = None
+
+
+ def __str__(self):
+ if not self.fsthandle:
+ raise ValueError('FSM not defined')
+ foma_fsm_count(self.fsthandle)
+ s = 'Name: %s\n' % self.fsthandle.contents.name
+ s += 'States: %i\n' % self.fsthandle.contents.statecount
+ s += 'Transitions: %i\n' % self.fsthandle.contents.arccount
+ s += 'Final states: %i\n' % self.fsthandle.contents.finalcount
+ s += 'Deterministic: %i\n' % self.fsthandle.contents.is_deterministic
+ s += 'Minimized: %i\n' % self.fsthandle.contents.is_minimized
+ s += 'Numtapes: %i\n' % self.numtapes
+ return s
+
+ def parse(self, word):
+ #[word/□ .o. [0:?^(numtapes-1) ?]*].l & Grammar ;
+ m = self.numtapes
+ regx = (u'[{' + word + u'}/□ .o. [0:?^' + str(m-1) + u' ?]*].l')
+ reg = FST(regx)
+ gr = FST()
+ gr.fsthandle = foma_fsm_copy(self.fsthandle)
+ res = MTFSM(reg.intersect(gr), numtapes = m)
+ return res
+
+ def join(self, other):
+ """Joins two multitape FSMs by composition. E.g.
+ [ a d ] [ c □ □ ] [ a d □ ]
+ A = [ b e ] B = [ d f g ], and A.join(B) = [ b e □ ]
+ [ c □ ] [ e f g ] [ c □ □ ]
+ [ d f g ]
+ [ e f g ] """
+
+ m = self.numtapes
+ n = other.numtapes
+ pada = FST('[0:□^' + str(m) +' [0:?^' + str(n-1) + ' - 0:□^' + str(n-1) + '] | ?^' + str(m) + ' 0:?^' + str(n-1) + ']*')
+ padb = FST('[[0:?^' + str(m-1) + ' - 0:□^' + str(m-1) + ' ] 0:□^' + str(n) + '| 0:?^' + str(m-1) + ' ?^' + str(n) + ']*')
+ extenda = self.compose(pada).lower()
+ extendb = other.compose(padb).lower()
+ flt = FST('~[?^' + str(m+n-1) +'* [□^' + str(m) + ' ?^' + str(n-1) + ' [?^' + str(m-1) + ' □^' + str(n) + ' |[?^' + str(m-1) +' - □^' + str(m-1) + ' ] □ [?^' + str(n-1) + ' - □^' + str(n-1) + ' ]] | ?^' + str(m-1) + ' □^' + str(n) + ' [□^' + str(m) + ' ?^' + str(n-1) + ' |[?^' + str(m-1) + ' - □^' + str(m-1) + ' ] □ [?^' + str(n-1) + ' - □^' + str(n-1) + ' ]]] ?*]')
+ res = extenda & extendb & flt
+ result = MTFSM(res, m + n - 1)
+ return result
+
+ def _apply(self, applyf, word = None):
+ if not self.fsthandle:
+ raise ValueError('FST not defined')
+ applyerhandle = foma_apply_init(self.fsthandle)
+ output = applyf(c_void_p(applyerhandle))
+ while True:
+ if output == None:
+ foma_apply_clear(c_void_p(applyerhandle))
+ return
+ else:
+ yield output
+ if word:
+ output = applyf(c_void_p(applyerhandle), None)
+ else:
+ output = applyf(c_void_p(applyerhandle))
+
+ def __iter__(self):
+ return self._mtwords()
+
+ def __add__(self, other):
+ return self.join(other)
+
+ def _fmt(self, word):
+ cols = word
+ colchunks = [map(lambda z: len(z), word[x:x+self.numtapes]) for x in xrange(0, len(word), self.numtapes)]
+ col_widths = [max(x) for x in colchunks]
+ format = ' '.join(['%%-%ds' % width for width in col_widths])
+ # string to rows
+ rows = [[word[y] for y in xrange(x, len(word), self.numtapes)] for x in xrange(self.numtapes)]
+ s = ''
+ for row in rows:
+ #s += format % tuple(row) + '\n'
+ s += ''.join(row) + '\n'
+ return s
+
+ def _mtwords(self):
+ applyf = foma_apply_upper_words
+ if not self.fsthandle:
+ raise ValueError('FST not defined')
+ applyerhandle = foma_apply_init(self.fsthandle)
+ toksym = '\x07'
+ foma_apply_set_space_symbol(c_void_p(applyerhandle), c_char_p(toksym))
+ output = applyf(c_void_p(applyerhandle))
+ while True:
+ if output == None:
+ foma_apply_clear(c_void_p(applyerhandle))
+ return
+ else:
+ yield self._fmt(output[:-1].split('\x07'))
+ output = applyf(c_void_p(applyerhandle))
diff --git a/regex.l b/regex.l
new file mode 100644
index 0000000..0b6ec4d
--- /dev/null
+++ b/regex.l
@@ -0,0 +1,461 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+%option noinput
+%option reentrant bison-bridge
+%option bison-locations
+%option yylineno
+%{
+#include <stdio.h>
+#include "foma.h"
+#include "regex.h"
+
+struct defs {
+ struct defined_networks *defined_nets;
+ struct defined_functions *defined_funcs;
+};
+
+#define YY_EXTRA_TYPE struct defs *
+
+#define YY_USER_ACTION yylloc->first_line = yylloc->last_line = yylineno; yylloc->first_column = yycolumn; yylloc->last_column = yycolumn+yyleng-1; yycolumn += yyleng;
+
+#define MAX_PARSE_DEPTH 100
+
+struct parser_vars {
+ int rewrite;
+ int rule_direction;
+ struct fsmcontexts *contexts;
+ struct fsmrules *rules;
+ struct rewrite_set *rewrite_rules;
+ char *ystring;
+ struct fsm *ynet;
+ int ytype;
+};
+
+struct parser_vars parservarstack[MAX_PARSE_DEPTH];
+int g_parse_depth = 0;
+
+extern int yyparse();
+extern int get_iface_lineno(void);
+extern int rewrite, rule_direction, substituting;
+extern struct fsmcontexts *contexts;
+extern struct fsmrules *rules;
+extern struct rewrite_set *rewrite_rules;
+extern struct fsm *current_parse;
+
+char *yyget_text(yyscan_t yyscanner);
+char *tempstr, *tempstr2;
+int yylex_init (yyscan_t* scanner);
+int yylex_init_extra (struct defs *defptr, yyscan_t *scanner);
+int yylex_destroy (yyscan_t scanner);
+int my_yyparse(char *my_string, int lineno, struct defined_networks *defined_nets, struct defined_functions *defined_funcs);
+
+int yywrap(yyscan_t scanner) {return 1; }
+
+int yyerror(YYLTYPE* yylloc, yyscan_t scanner, char *msg) {
+ if(yylloc->first_line)
+ fprintf(stderr, "%d.%d-%d.%d: error: ", yylloc->first_line, yylloc->first_column, yylloc->last_line, yylloc->last_column);
+ fprintf(stderr, "%s%s at '%s'.\n", "***", msg, yyget_text(scanner));
+ return 1;
+}
+
+struct fsm *fsm_parse_regex(char *regex, struct defined_networks *defined_nets, struct defined_functions *defined_funcs) {
+ char *newregex;
+ current_parse = NULL;
+ newregex = xxmalloc(sizeof(char)*(strlen(regex)+2));
+ strcpy(newregex, regex);
+ strcat(newregex, ";");
+ if (my_yyparse(newregex, 1, defined_nets, defined_funcs) == 0) {
+ xxfree(newregex);
+ return(fsm_minimize(current_parse));
+ } else {
+ xxfree(newregex);
+ return(NULL);
+ }
+}
+
+/* Only used when reading regex from file */
+struct fsm *fsm_parse_regex_string(char *regex) {
+ current_parse = NULL;
+ if (my_yyparse(regex,1,g_defines,g_defines_f) == 0) {
+ xxfree(regex);
+ return(fsm_minimize(current_parse));
+ } else {
+ xxfree(regex);
+ return(NULL);
+ }
+}
+
+void yyset_lineno(int line_number, yyscan_t yyscanner);
+
+int my_yyparse(char *my_string, int lineno, struct defined_networks *defined_nets, struct defined_functions *defined_funcs) {
+ int yyp;
+ yyscan_t scanner;
+ struct defs defsptr[1];
+ YY_BUFFER_STATE my_string_buffer;
+
+ defsptr->defined_nets = defined_nets;
+ defsptr->defined_funcs = defined_funcs;
+ yylex_init_extra(defsptr, &scanner);
+
+ my_string_buffer = yy_scan_string(my_string, scanner);
+ yyset_lineno(lineno, scanner);
+ if (g_parse_depth > 0) {
+ if (g_parse_depth >= MAX_PARSE_DEPTH) {
+ fprintf(stderr,"Exceeded parser stack depth. Self-recursive call?\n");
+ return 1;
+ }
+ /* Save variables on stack */
+ parservarstack[g_parse_depth].rewrite = rewrite;
+ parservarstack[g_parse_depth].rule_direction = rule_direction;
+ parservarstack[g_parse_depth].contexts = contexts;
+ parservarstack[g_parse_depth].rules = rules;
+ parservarstack[g_parse_depth].rewrite_rules = rewrite_rules;
+ }
+ g_parse_depth++;
+ yyp = yyparse(scanner, defined_nets, defined_funcs);
+ g_parse_depth--;
+ if (g_parse_depth > 0) {
+ /* Restore parse variables */
+ rewrite = parservarstack[g_parse_depth].rewrite;
+ rule_direction = parservarstack[g_parse_depth].rule_direction;
+ contexts = parservarstack[g_parse_depth].contexts;
+ rules = parservarstack[g_parse_depth].rules;
+ rewrite_rules = parservarstack[g_parse_depth].rewrite_rules;
+ }
+ yy_delete_buffer(my_string_buffer, scanner);
+ yylex_destroy(scanner);
+ return yyp;
+}
+
+%}
+
+ANYUTF [\001-\177]|[\300-\337].|[\340-\357]..|[\360-\367]...
+ /* Reserved multicharacter symbols are a little tricky to define */
+ /* what we're doing is excluding some combinations of multibyte sequences */
+ /* using the {-} construct in flex */
+NONRESERVED [0-9A-Za-z\?\'\=]|[\300-\301].|[\302]([\000-\377]{-}[\254])|[\303]([\000-\377]{-}[\227])|[\304-\315].|[\316]([\000-\377]{-}[\243\265])|[\317-\337].|[\340-\341]..|[\342][\000-\200].|[\342][\201][\000-\377]{-}[\273]|[\342][\202][\000-\377]{-}[\201\202]|[\342][\203-\205][\000-\377]|[\342][\206]([\000-\377]{-}[\222\224])|[\342][\207].|[\342][\210]([\000-\377]{-}[\200\203\205\210\230\245\247\250\251\252])|[\342][\211]([\000-\377]{-}[\240\244\245\272\273])|[\342][\212-\377].|[\343- [...]
+BRACED [{]([^}]|[\300-\337].|[\340-\357]..|[\360-\367]...)+[}]
+
+%x DEFI QTD QTDEND UQ EQ ENDQ
+
+%%
+
+ /* we're matching braced strings */
+
+{BRACED} {
+ yylval_param->net = fsm_explode(yytext);
+ return NET;
+}
+
+ /* Read binary file */
+([\100]["][^"]+[\042]) {
+ tempstr = xxstrndup(yytext+2,yyleng-3);
+ yylval_param->net = fsm_read_binary_file(tempstr);
+ xxfree(tempstr);
+ if (yylval_param->net != NULL) {
+ return NET;
+ }
+}
+
+ /* Read regex from file */
+([\100]re["][^"]+[\042]) {
+ tempstr = xxstrndup(yytext+4,yyleng-5);
+ tempstr2 = file_to_mem(tempstr);
+ xxfree(tempstr);
+ if (tempstr2 != NULL) {
+ yylval_param->net = fsm_parse_regex_string(tempstr2);
+ if (yylval_param->net != NULL) {
+ return NET;
+ }
+ }
+}
+
+ /* Read text file */
+([\100]txt["][^"]+[\042]) {
+ tempstr = xxstrndup(yytext+5,yyleng-6);
+ yylval_param->net = fsm_read_text_file(tempstr);
+ xxfree(tempstr);
+ if (yylval_param->net != NULL) {
+ return NET;
+ }
+}
+
+ /* Read spaced text file */
+([\100]stxt["][^"]+[\042]) {
+ tempstr = xxstrndup(yytext+6,yyleng-7);
+ yylval_param->net = fsm_read_spaced_text_file(tempstr);
+ xxfree(tempstr);
+ if (yylval_param->net != NULL) {
+ return NET;
+ }
+}
+
+(\^[0-9]+) {
+ //yylval_param->string = xxstrdup(yytext+1);
+ yylval_param->string = yytext+1;
+ return NCONCAT;
+}
+
+(\^\{[0-9]+[ ]*,[ ]*[0-9]+\}) {
+ //yylval_param->string = xxstrdup(yytext+2);
+ yylval_param->string = yytext+2;
+ return MNCONCAT;
+}
+(\^\>[0-9]+) {
+ yylval_param->string = yytext+2;
+// yylval_param->string = xxstrdup(yytext+2);
+ return MORENCONCAT;
+}
+(\^\<[0-9]+) {
+ // yylval_param->string = xxstrdup(yytext+2);
+ yylval_param->string = yytext+2;
+ return LESSNCONCAT;
+}
+
+((\ |\t)+) {
+
+}
+
+ /* Start of universal quantifier */
+[\050][\342][\210][\200]/(([\000-\177]{-}[\051])|[\300-\337].|[\340-\357]..|[\360-\367]...)+[\051] {
+ BEGIN(UQ);
+}
+
+ /* Start of existential quantifier */
+[\050][\342][\210][\203]/(([\000-\177]{-}[\051])|[\300-\337].|[\340-\357]..|[\360-\367]...)+[\051] {
+ BEGIN(EQ);
+}
+
+
+<UQ>(([\000-\177]{-}[\051])|[\300-\337].|[\340-\357]..|[\360-\367]...)+/[\051] {
+ /* Add quantifier to quantifier symbol table */
+
+ //yylval_param->string = xxstrdup(yytext);
+ yylval_param->string = yytext;
+ add_quantifier(yytext);
+ BEGIN(ENDQ);
+ return(UQUANT);
+}
+
+<EQ>(([\000-\177]{-}[\051])|[\300-\337].|[\340-\357]..|[\360-\367]...)+/[\051] {
+ /* Add quantifier to quantifier symbol table */
+ //yylval_param->string = xxstrdup(yytext);
+ yylval_param->string = yytext;
+ add_quantifier(yytext);
+ BEGIN(ENDQ);
+ return(EQUANT);
+}
+
+<ENDQ>[\051] {BEGIN(INITIAL);}
+
+ /* Start of a quoted sequence of symbols */
+[\042]/([\300-\337].|[\340-\357]..|[\360-\367]...|[\001-\041]|[\043-\177])+[\042] {
+ BEGIN(QTD);
+}
+
+ /* Stuff that goes inside " ", including UTF8 \uHHHH sequences */
+<QTD>([\300-\337].|[\340-\357]..|[\360-\367]...|[\001-\041]|[\043-\177])+/[\042] {
+ decode_quoted(yytext);
+ yylval_param->net = fsm_symbol(yytext);
+ BEGIN(QTDEND);
+ return NET;
+}
+
+ /* Disregard end quote */
+<QTDEND>[\042] { BEGIN(INITIAL);}
+
+ /* Different epsilon variants: "" or [] or \epsilon */
+[\042][\042]|\[\]|[\316][\265] {
+ yylval_param->net = fsm_empty_string();
+ return NET;
+}
+ /* The empty set */
+[\342][\210][\205] {
+ yylval_param->net = fsm_empty_set();
+ return NET;
+}
+
+ /* Sigma */
+[\316][\243] {
+ yylval_param->net = fsm_identity();
+ return NET;
+}
+
+(_S\() { return SUCCESSOR_OF; }
+
+(_isunambiguous\() { return ISUNAMBIGUOUS; }
+(_isidentity\() { return ISIDENTITY; }
+(_isfunctional\() { return ISFUNCTIONAL; }
+(_notid\() { return NOTID; }
+(_lm\() { return LETTERMACHINE; }
+(_loweruniq\() { return LOWERUNIQ; }
+(_loweruniqeps\() { return LOWERUNIQEPS; }
+(_allfinal\() { return ALLFINAL; }
+(_unambpart\() { return UNAMBIGUOUSPART; }
+(_ambpart\() { return AMBIGUOUSPART; }
+(_ambdom\() { return AMBIGUOUSDOMAIN; }
+(_eq\() { return EQSUBSTRINGS; }
+(_marktail\() { return MARKFSMTAIL; }
+(_addfinalloop\() { return MARKFSMTAILLOOP; }
+(_addnonfinalloop\() { return MARKFSMMIDLOOP; }
+(_addloop\() { return MARKFSMLOOP; }
+(_addsink\() { return ADDSINK; }
+(_leftrewr\() { return LEFTREWR; }
+(_flatten\() { return FLATTEN; }
+(_sublabel\() { return SUBLABEL; }
+(_close\() { return CLOSESIGMA; }
+(_closeu\() { return CLOSESIGMAUNK; }
+
+({NONRESERVED}+\() {
+ yylval_param->string = xxstrdup(yytext);
+ //yylval_param->string = yytext;
+ return FUNCTION;
+}
+
+ /* The set of nonreserved symbols, or % followed by any UTF8 character */
+
+({NONRESERVED}|\%{ANYUTF})+ {
+ int i,j, skip, escaped;
+ if ((strncmp(yytext,"=",1) == 0) && (count_quantifiers() > 0)) {
+ int i;
+ /* Copy yytext because unput() trashes yytext */
+ char *yycopy = xxstrdup(yytext);
+ for ( i = yyleng - 1; i > 0; --i )
+ unput( yycopy[i] );
+ xxfree(yycopy);
+ return EQUALS;
+ }
+
+ for (escaped=0, i=0,j=0;*(yytext+i);) {
+ *(yytext+j) = *(yytext+i);
+ if (*(yytext+i) == '%') /* Skip escaping percent sign */ {
+ i++;
+ escaped++;
+ }
+ for(skip = utf8skip(yytext+i)+1; skip > 0; skip--) {
+ *(yytext+j) = *(yytext+i);
+ i++; j++;
+ }
+ }
+ *(yytext+j) = *(yytext+i);
+ if (substituting) {
+ yylval_param->string = xxstrdup(yytext);
+ //yylval_param->string = yytext;
+ return SUBVAL;
+ }
+ // yylval_param->string = xxstrdup(yytext);
+ yylval_param->string = yytext;
+ if(find_defined(yyextra->defined_nets, yytext) != NULL) {
+ yylval_param->net = fsm_copy(find_defined(yyextra->defined_nets, yytext));
+ } else if (find_quantifier(yytext) != NULL) {
+ return VAR;
+ } else {
+ if (!escaped && strcmp(yytext, "0") == 0)
+ yylval_param->net = fsm_empty_string();
+ else if (!escaped && strcmp(yytext, "?") == 0)
+ yylval_param->net = fsm_identity();
+ else
+ yylval_param->net = fsm_symbol(yytext);
+ }
+ return NET;
+}
+
+\.#\. { yylval_param->net = fsm_symbol(".#."); return NET; }
+(\_) { return CONTEXT; }
+\.(u|1)|[\342][\202][\201] { return XUPPER; }
+\.(l|2)|[\342][\202][\202] { return XLOWER; }
+(\.f) { return FLAG_ELIMINATE; }
+\[\./[^#] { return LDOT; }
+(\.\]) { return RDOT; }
+([\342][\206][\222]) { return IMPLIES; }
+([\342][\206][\224]) { return BICOND; }
+([\xe2][\x88][\x88]) { return IN; }
+(\~|[\302][\254]) { return COMPLEMENT; }
+(\.o\.|[\342][\210][\230]) { return COMPOSE; }
+(\.O\.) { return LENIENT_COMPOSE; }
+(\.P\.) { return PRIORITY_UNION_U; }
+(\.p\.) { return PRIORITY_UNION_L; }
+(<>|[\342][\210][\245]) { return SHUFFLE; }
+(<) { return PRECEDES; }
+(>) { return FOLLOWS; }
+(\.\.\.) { return TRIPLE_DOT; }
+(\-\>) { yylval_param->type = ARROW_RIGHT; return ARROW;}
+(\(\-\>\)) { yylval_param->type = ARROW_RIGHT | ARROW_OPTIONAL; return ARROW;}
+(\<\-) { yylval_param->type = ARROW_LEFT; return ARROW;}
+(\(\<\-\)) { yylval_param->type = ARROW_LEFT | ARROW_OPTIONAL; return ARROW;}
+(\<\-\>) { yylval_param->type = ARROW_LEFT|ARROW_RIGHT; return ARROW;}
+(\(\<\-\>\)) { yylval_param->type = ARROW_LEFT|ARROW_RIGHT|ARROW_OPTIONAL; return ARROW;}
+(@\-\>) { yylval_param->type = ARROW_RIGHT|ARROW_LONGEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(\(@\-\>\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_RIGHT|ARROW_LONGEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(@\>) { yylval_param->type = ARROW_RIGHT|ARROW_SHORTEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(\(@\>\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_RIGHT|ARROW_SHORTEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(\-\>@) { yylval_param->type = ARROW_RIGHT|ARROW_LONGEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(\(\-\>@\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_RIGHT|ARROW_LONGEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(\>@) { yylval_param->type = ARROW_RIGHT|ARROW_SHORTEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(\(\>@\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_RIGHT|ARROW_SHORTEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(\<\-@) { yylval_param->type = ARROW_LEFT|ARROW_LONGEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(\(\<\-@\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_LEFT|ARROW_LONGEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(\<@) { yylval_param->type = ARROW_LEFT|ARROW_SHORTEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(\(\<@\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_LEFT|ARROW_SHORTEST_MATCH|ARROW_LEFT_TO_RIGHT; return ARROW;}
+(@\<\-) { yylval_param->type = ARROW_LEFT|ARROW_LONGEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(\(@\<\-\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_LEFT|ARROW_LONGEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(@\<) { yylval_param->type = ARROW_LEFT|ARROW_SHORTEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+(\(@\<\)) { yylval_param->type = ARROW_OPTIONAL|ARROW_RIGHT|ARROW_SHORTEST_MATCH|ARROW_RIGHT_TO_LEFT; return ARROW;}
+
+(\=\>) { return CRESTRICT; }
+([\140]) { return SUBSTITUTE; }
+(\|\|) { yylval_param->type = OP_UPWARD_REPLACE; return DIRECTION; }
+(\/\/) { yylval_param->type = OP_RIGHTWARD_REPLACE; return DIRECTION; }
+(\\\\) { yylval_param->type = OP_LEFTWARD_REPLACE; return DIRECTION; }
+(\\\/) { yylval_param->type = OP_DOWNWARD_REPLACE; return DIRECTION; }
+(\|\|\|) { yylval_param->type = OP_TWO_LEVEL_REPLACE; return DIRECTION; }
+(:) { return HIGH_CROSS_PRODUCT; }
+(\.x\.|[\303][\227]) { return CROSS_PRODUCT; }
+(,) { return COMMA; }
+(,,) { return DOUBLE_COMMA; }
+(\.\/\.) { return IGNORE_INTERNAL; }
+(\/) { return IGNORE_ALL; }
+(\/\/\/) { return RIGHT_QUOTIENT; }
+(\\\\\\) { return LEFT_QUOTIENT; }
+(\/\\\/) { return INTERLEAVE_QUOTIENT; }
+(\\) { return TERM_NEGATION; }
+(\-) { return MINUS; }
+(\$\?) { return CONTAINS_OPT_ONE; }
+(\$\.) { return CONTAINS_ONE; }
+(\$) { return CONTAINS; }
+(\+) { return KLEENE_PLUS; }
+(\*) { return KLEENE_STAR; }
+\.i|[\342][\201][\273][\302][\271] { return INVERSE; }
+(\.r) { return REVERSE; }
+(\[) { return LBRACKET; }
+(\]) { return RBRACKET; }
+[\342][\211][\272] { return PRECEDES; }
+[\342][\211][\273] { return FOLLOWS; }
+[\342][\211][\240] { return NEQ; }
+(\() { return LPAREN; }
+(\)) { return RPAREN; }
+(;m) { return ENDM; }
+(;d) { return ENDD; }
+(;) { return END; }
+(\||[\342][\210][\250]|[\342][\210][\252]) { return UNION; }
+(\&|[\342][\210][\247]|[\342][\210][\251]) { return INTERSECT; }
+
+((#|!).*\n) { yycolumn = 1; }
+(\n+) { }
+(\r+) { }
+(.) { }
diff --git a/regex.y b/regex.y
new file mode 100644
index 0000000..dfddbf0
--- /dev/null
+++ b/regex.y
@@ -0,0 +1,420 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+%{
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "foma.h"
+#define MAX_F_RECURSION 100
+extern int yyerror();
+extern int yylex();
+extern int my_yyparse(char *my_string, int lineno, struct defined_networks *defined_nets, struct defined_functions *defined_funcs);
+struct fsm *current_parse;
+int rewrite, rule_direction;
+int substituting = 0;
+static char *subval1, *subval2;
+struct fsmcontexts *contexts;
+struct fsmrules *rules;
+struct rewrite_set *rewrite_rules;
+static struct fsm *fargs[100][MAX_F_RECURSION]; /* Function arguments [number][frec] */
+static int frec = -1; /* Current depth of function recursion */
+static char *fname[MAX_F_RECURSION]; /* Function names */
+static int fargptr[MAX_F_RECURSION]; /* Current argument no. */
+/* Variable to produce internal symbols */
+unsigned int g_internal_sym = 23482342;
+
+void add_function_argument(struct fsm *net) {
+ fargs[fargptr[frec]][frec] = net;
+ fargptr[frec]++;
+}
+
+void declare_function_name(char *s) {
+ if (frec > MAX_F_RECURSION) {
+ printf("Function stack depth exceeded. Aborting.\n");
+ exit(1);
+ }
+ fname[frec] = xxstrdup(s);
+ xxfree(s);
+}
+
+struct fsm *function_apply(struct defined_networks *defined_nets, struct defined_functions *defined_funcs) {
+ int i, mygsym, myfargptr;
+ char *regex;
+ char repstr[13], oldstr[13];
+ if ((regex = find_defined_function(defined_funcs, fname[frec],fargptr[frec])) == NULL) {
+ fprintf(stderr, "***Error: function %s@%i) not defined!\n",fname[frec], fargptr[frec]);
+ return NULL;
+ }
+ regex = xxstrdup(regex);
+ mygsym = g_internal_sym;
+ myfargptr = fargptr[frec];
+ /* Create new regular expression from function def. */
+ /* and parse that */
+ for (i = 0; i < fargptr[frec]; i++) {
+ sprintf(repstr,"%012X",g_internal_sym);
+ sprintf(oldstr, "@ARGUMENT%02i@", (i+1));
+ streqrep(regex, oldstr, repstr);
+ /* We temporarily define a network and save argument there */
+ /* The name is a running counter g_internal_sym */
+ add_defined(defined_nets, fargs[i][frec], repstr);
+ g_internal_sym++;
+ }
+ my_yyparse(regex,1,defined_nets, defined_funcs);
+ for (i = 0; i < myfargptr; i++) {
+ sprintf(repstr,"%012X",mygsym);
+ /* Remove the temporarily defined network */
+ remove_defined(defined_nets, repstr);
+ mygsym++;
+ }
+ xxfree(fname[frec]);
+ frec--;
+ xxfree(regex);
+ return(current_parse);
+}
+
+void add_context_pair(struct fsm *L, struct fsm *R) {
+ struct fsmcontexts *newcontext;
+ newcontext = xxcalloc(1,sizeof(struct fsmcontexts));
+ newcontext->left = L;
+ newcontext->right = R;
+ newcontext->next = contexts;
+ contexts = newcontext;
+}
+
+void clear_rewrite_ruleset(struct rewrite_set *rewrite_rules) {
+ struct rewrite_set *rule, *rulep;
+ struct fsmcontexts *contexts, *contextsp;
+ struct fsmrules *r, *rp;
+ for (rule = rewrite_rules; rule != NULL; rule = rulep) {
+
+ for (r = rule->rewrite_rules ; r != NULL; r = rp) {
+ fsm_destroy(r->left);
+ fsm_destroy(r->right);
+ fsm_destroy(r->right2);
+ fsm_destroy(r->cross_product);
+ rp = r->next;
+ xxfree(r);
+ }
+
+ for (contexts = rule->rewrite_contexts; contexts != NULL ; contexts = contextsp) {
+
+ contextsp = contexts->next;
+ fsm_destroy(contexts->left);
+ fsm_destroy(contexts->right);
+ fsm_destroy(contexts->cpleft);
+ fsm_destroy(contexts->cpright);
+ xxfree(contexts);
+ }
+ rulep = rule->next;
+ //fsm_destroy(rules->cpunion);
+ xxfree(rule);
+ }
+}
+
+void add_rewrite_rule() {
+ struct rewrite_set *new_rewrite_rule;
+ if (rules != NULL) {
+ new_rewrite_rule = xxmalloc(sizeof(struct rewrite_set));
+ new_rewrite_rule->rewrite_rules = rules;
+ new_rewrite_rule->rewrite_contexts = contexts;
+ new_rewrite_rule->next = rewrite_rules;
+ new_rewrite_rule->rule_direction = rule_direction;
+
+ rewrite_rules = new_rewrite_rule;
+ rules = NULL;
+ contexts = NULL;
+ rule_direction = 0;
+ }
+}
+
+void add_eprule(struct fsm *R, struct fsm *R2, int type) {
+ struct fsmrules *newrule;
+ rewrite = 1;
+ newrule = xxmalloc(sizeof(struct fsmrules));
+ newrule->left = fsm_empty_string();
+ newrule->right = R;
+ newrule->right2 = R2;
+ newrule->arrow_type = type;
+ newrule->next = rules;
+ rules = newrule;
+}
+
+void add_rule(struct fsm *L, struct fsm *R, struct fsm *R2, int type) {
+ struct fsm *test;
+ struct fsmrules *newrule;
+ rewrite = 1;
+ newrule = xxmalloc(sizeof(struct fsmrules));
+
+ if ((type & ARROW_DOTTED) != 0) {
+ newrule->left = fsm_minus(fsm_copy(L), fsm_empty_string());
+ } else {
+ newrule->left = L;
+ }
+ newrule->right = R;
+ newrule->right2 = R2;
+ newrule->next = rules;
+ newrule->arrow_type = type;
+ if ((type & ARROW_DOTTED) != 0) {
+ newrule->arrow_type = type - ARROW_DOTTED;
+ }
+
+ rules = newrule;
+
+ if ((type & ARROW_DOTTED) != 0) {
+ /* Add empty [..] -> B for dotted rules (only if LHS contains the empty string) */
+ test = fsm_intersect(L,fsm_empty_string());
+ if (!fsm_isempty(test)) {
+ newrule = xxmalloc(sizeof(struct fsmrules));
+ newrule->left = test;
+ newrule->right = fsm_copy(R);
+ newrule->right2 = fsm_copy(R2);
+ newrule->next = rules;
+ newrule->arrow_type = type;
+ rules = newrule;
+ } else {
+ //fsm_destroy(test);
+ }
+ }
+}
+
+
+%}
+
+%union {
+ char *string;
+ struct fsm *net;
+ int type;
+}
+
+%pure-parser
+%expect 686
+%parse-param { void *scanner }
+%parse-param { struct defined_networks *defined_nets }
+%parse-param { struct defined_functions *defined_funcs } /* Assume yyparse is called with this argument */
+%lex-param { yyscan_t *scanner } /* Call flex functions with this argument */
+%locations
+%initial-action {
+ clear_quantifiers();
+ rewrite = 0;
+ contexts = NULL;
+ rules = NULL;
+ rewrite_rules = NULL;
+ rule_direction = 0;
+ substituting = 0;
+};
+
+/* precedence
+ \ ` Term complement, Substitution
+ : High-precedence crossproduct operator
+ + * ^ .1 .2 .u .l .i .r Kleene plus and star, iteration, upper-lower, invert and reverse
+ ~ $ $. $? Complement, containments
+ / /// \\\ /\/ Ignore, Quotients
+ Concatenation (no overt operator)
+ > < Precede and follow
+ | & - .P. .p. Union, intersect, minus, priority unions
+ => -> (->) @-> etc. Rule operators
+ <> Shuffle
+ _ args
+ .x. .o. .O. Crossproduct and composes
+*/
+
+%token <net> NET
+%token <string> END LBRACKET RBRACKET LPAREN RPAREN ENDM ENDD CRESTRICT CONTAINS CONTAINS_OPT_ONE CONTAINS_ONE XUPPER XLOWER FLAG_ELIMINATE IGNORE_ALL IGNORE_INTERNAL CONTEXT NCONCAT MNCONCAT MORENCONCAT LESSNCONCAT DOUBLE_COMMA COMMA SHUFFLE PRECEDES FOLLOWS RIGHT_QUOTIENT LEFT_QUOTIENT INTERLEAVE_QUOTIENT UQUANT EQUANT VAR IN IMPLIES BICOND EQUALS NEQ SUBSTITUTE SUCCESSOR_OF PRIORITY_UNION_U PRIORITY_UNION_L LENIENT_COMPOSE TRIPLE_DOT LDOT RDOT FUNCTION SUBVAL ISUNAMBIGUOUS ISIDENTITY [...]
+
+%token <type> ARROW DIRECTION
+
+%type <net> network networkA n0 network1 network2 network3 network4 network5 network6 network7 network8 network9 network10 network11 network12 fstart fmid fend sub1 sub2
+
+%left COMPOSE CROSS_PRODUCT HIGH_CROSS_PRODUCT COMMA SHUFFLE PRECEDES FOLLOWS LENIENT_COMPOSE
+%left UNION INTERSECT MINUS
+%left COMPLEMENT
+%left KLEENE_STAR KLEENE_PLUS REVERSE INVERSE
+%left TERM_NEGATION
+
+/* Regular expression grammar */
+%%
+
+start: regex
+| regex start
+
+regex:
+network END { current_parse = $1; }
+
+network: networkA { }
+| network COMPOSE networkA { $$ = fsm_compose($1,$3); }
+| network LENIENT_COMPOSE networkA { $$ = fsm_lenient_compose($1,$3); }
+| network CROSS_PRODUCT networkA { $$ = fsm_cross_product($1,$3); }
+
+networkA: n0 { if (rewrite) { add_rewrite_rule(); $$ = fsm_rewrite(rewrite_rules); clear_rewrite_ruleset(rewrite_rules); } rewrite = 0; contexts = NULL; rules = NULL; rewrite_rules = NULL; }
+
+n0: network1 { }
+| n0 CONTEXT n0 { $$ = NULL; add_context_pair($1,$3);}
+| n0 CONTEXT { add_context_pair($1,fsm_empty_string()); }
+| n0 CONTEXT COMMA n0 { add_context_pair($1,fsm_empty_string()); }
+| CONTEXT { add_context_pair(fsm_empty_string(),fsm_empty_string());}
+| CONTEXT DOUBLE_COMMA n0 { add_rewrite_rule(); add_context_pair(fsm_empty_string(),fsm_empty_string());}
+| n0 CONTEXT DOUBLE_COMMA n0 { add_rewrite_rule(); add_context_pair($1,fsm_empty_string());}
+| CONTEXT COMMA n0 { add_context_pair(fsm_empty_string(),fsm_empty_string());}
+| CONTEXT n0 { add_context_pair(fsm_empty_string(),$2); }
+| CONTEXT n0 COMMA n0 { add_context_pair(fsm_empty_string(),$2); }
+| n0 CONTEXT n0 COMMA n0 { add_context_pair($1,$3); }
+| n0 CRESTRICT n0 { $$ = fsm_context_restrict($1,contexts); fsm_clear_contexts(contexts);}
+| n0 ARROW n0 { add_rule($1,$3,NULL,$2); if ($1->arity == 2) { printf("Error: LHS is transducer\n"); YYERROR;}}
+| n0 ARROW { add_rule($1,NULL,NULL,$2); }
+
+| LDOT n0 RDOT ARROW n0 { add_rule($2,$5,NULL,$4|ARROW_DOTTED); if ($5 == NULL) { YYERROR;}}
+| LDOT RDOT ARROW n0 { add_eprule($4,NULL,$3|ARROW_DOTTED); if ($4 == NULL) { YYERROR;}}
+| LDOT n0 RDOT ARROW n0 COMMA n0 { add_rule($2,$5,NULL,$4|ARROW_DOTTED);}
+| LDOT RDOT ARROW n0 COMMA n0 { add_eprule($4,NULL,$3|ARROW_DOTTED);}
+| LDOT n0 RDOT ARROW n0 DIRECTION n0 { add_rule($2,$5,NULL,$4|ARROW_DOTTED); rule_direction = $6;}
+| LDOT RDOT ARROW n0 DIRECTION n0 { add_eprule($4,NULL,$3|ARROW_DOTTED); rule_direction = $5;}
+| n0 ARROW n0 COMMA n0 { add_rule($1,$3,NULL,$2);}
+| n0 ARROW COMMA n0 { add_rule($1,NULL,NULL,$2);}
+| n0 ARROW n0 DIRECTION n0 { add_rule($1,$3,NULL,$2); rule_direction = $4;}
+| n0 ARROW DIRECTION n0 { add_rule($1,NULL,NULL,$2); rule_direction = $3;}
+
+| n0 DOUBLE_COMMA n0 { add_rewrite_rule();}
+
+| n0 ARROW TRIPLE_DOT { add_rule($1,fsm_empty_string(),fsm_empty_string(),$2);}
+| n0 ARROW TRIPLE_DOT n0 { add_rule($1,fsm_empty_string(),$4,$2);}
+| n0 ARROW n0 TRIPLE_DOT { add_rule($1,$3,fsm_empty_string(),$2);}
+| n0 ARROW n0 TRIPLE_DOT n0 { add_rule($1,$3,$5,$2);}
+| n0 ARROW n0 TRIPLE_DOT n0 COMMA n0 { add_rule($1,$3,$5,$2);}
+| n0 ARROW n0 TRIPLE_DOT COMMA n0 { add_rule($1,$3,fsm_empty_string(),$2);}
+| n0 ARROW TRIPLE_DOT n0 COMMA n0 { add_rule($1,fsm_empty_string(),$4,$2);}
+| n0 ARROW TRIPLE_DOT COMMA n0 { add_rule($1,fsm_empty_string(),fsm_empty_string(),$2);}
+| n0 ARROW n0 TRIPLE_DOT n0 DIRECTION n0 { add_rule($1,$3,$5,$2); rule_direction = $6;}
+| n0 ARROW TRIPLE_DOT n0 DIRECTION n0 { add_rule($1,fsm_empty_string(),$4,$2); rule_direction = $5;}
+| n0 ARROW n0 TRIPLE_DOT DIRECTION n0 { add_rule($1,$3,fsm_empty_string(),$2); rule_direction = $5;}
+| n0 ARROW TRIPLE_DOT DIRECTION n0 { add_rule($1,fsm_empty_string(),fsm_empty_string(),$2); rule_direction = $4;}
+
+
+network1: network2 { }
+| network1 SHUFFLE network2 { $$ = fsm_shuffle($1,$3); }
+| network1 PRECEDES network2 { $$ = fsm_precedes($1,$3); }
+| network1 FOLLOWS network2 { $$ = fsm_follows($1,$3); }
+
+network2: network3 { };
+
+network3: network4 { };
+
+network4: network5 { }
+| network4 UNION network5 { $$ = fsm_union($1,$3); }
+| network4 PRIORITY_UNION_U network5 { $$ = fsm_priority_union_upper($1,$3); }
+| network4 PRIORITY_UNION_L network5 { $$ = fsm_priority_union_lower($1,$3); }
+| network4 INTERSECT network5 { $$ = fsm_intersect($1,$3); }
+| network4 MINUS network5 { $$ = fsm_minus($1,$3); }
+| network4 IMPLIES network5 { $$ = fsm_union(fsm_complement($1),$3); }
+| network4 BICOND network5 { $$ = fsm_intersect(fsm_union(fsm_complement(fsm_copy($1)),fsm_copy($3)), fsm_union(fsm_complement(fsm_copy($3)),fsm_copy($1))); fsm_destroy($1); fsm_destroy($3);}
+
+network5: network6 { }
+| network5 network6 { $$ = fsm_concat($1,$2); }
+| VAR IN network5 { $$ = fsm_ignore(fsm_contains(fsm_concat(fsm_symbol($1),fsm_concat($3,fsm_symbol($1)))),union_quantifiers(),OP_IGNORE_ALL); }
+
+| VAR EQUALS VAR { $$ = fsm_logical_eq($1,$3); }
+| VAR NEQ VAR { $$ = fsm_complement(fsm_logical_eq($1,$3)); }
+| VAR PRECEDES VAR { $$ = fsm_logical_precedence($1,$3); }
+| VAR FOLLOWS VAR { $$ = fsm_logical_precedence($3,$1); }
+
+network6: network7 { }
+| network6 IGNORE_ALL network7 { $$ = fsm_ignore($1,$3, OP_IGNORE_ALL); }
+| network6 IGNORE_INTERNAL network7 { $$ = fsm_ignore($1,$3, OP_IGNORE_INTERNAL); }
+| network6 RIGHT_QUOTIENT network7 { $$ = fsm_quotient_right($1,$3); }
+| network6 LEFT_QUOTIENT network7 { $$ = fsm_quotient_left($1,$3); }
+| network6 INTERLEAVE_QUOTIENT network7 { $$ = fsm_quotient_interleave($1,$3); }
+
+network7: network8 { }
+| COMPLEMENT network7 { $$ = fsm_complement($2); }
+| CONTAINS network7 { $$ = fsm_contains($2); }
+| CONTAINS_ONE network7 { $$ = fsm_contains_one($2); }
+| CONTAINS_OPT_ONE network7 { $$ = fsm_contains_opt_one($2); }
+
+network8: network9 { }
+
+network9: network10 { }
+| network9 KLEENE_STAR { $$ = fsm_kleene_star(fsm_minimize($1)); }
+| network9 KLEENE_PLUS { $$ = fsm_kleene_plus($1); }
+| network9 REVERSE { $$ = fsm_determinize(fsm_reverse($1)); }
+| network9 INVERSE { $$ = fsm_invert($1); }
+| network9 XUPPER { $$ = fsm_upper($1); }
+| network9 XLOWER { $$ = fsm_lower($1); }
+| network9 FLAG_ELIMINATE { $$ = flag_eliminate($1, NULL); }
+| network9 HIGH_CROSS_PRODUCT network10 { $$ = fsm_cross_product($1,$3); }
+
+| network9 NCONCAT { $$ = fsm_concat_n($1,atoi($2)); }
+| network9 MORENCONCAT { $$ = fsm_concat(fsm_concat_n(fsm_copy($1), atoi($2)),fsm_kleene_plus(fsm_copy($1))); fsm_destroy($1); }
+| network9 LESSNCONCAT { $$ = fsm_concat_m_n($1,0,atoi($2)-1); }
+| network9 MNCONCAT { $$ = fsm_concat_m_n($1,atoi($2),atoi(strstr($2,",")+1)); }
+
+network10: network11 { }
+| TERM_NEGATION network10 { $$ = fsm_term_negation($2); }
+
+network11: NET { $$ = $1;}
+| network12 { $$ = $1; }
+| UQUANT LPAREN network RPAREN { $$ = fsm_complement(fsm_substitute_symbol(fsm_intersect(fsm_quantifier($1),fsm_complement($3)),$1,"@_EPSILON_SYMBOL_@")); purge_quantifier($1); }
+| EQUANT network { $$ = fsm_substitute_symbol(fsm_intersect(fsm_quantifier($1),$2),$1,"@_EPSILON_SYMBOL_@"); purge_quantifier($1); }
+| LPAREN network RPAREN { if (count_quantifiers()) $$ = $2; else {$$ = fsm_optionality($2);} }
+| LBRACKET network RBRACKET { $$ = $2; }
+| SUCCESSOR_OF VAR COMMA VAR RPAREN {$$ = fsm_concat(fsm_universal(),fsm_concat(fsm_symbol($2),fsm_concat(fsm_universal(),fsm_concat(fsm_symbol($2),fsm_concat(union_quantifiers(),fsm_concat(fsm_symbol($4),fsm_concat(fsm_universal(),fsm_concat(fsm_symbol($4),fsm_universal())))))))); }
+| SUCCESSOR_OF VAR COMMA network RPAREN {$$ = fsm_concat(fsm_universal(),fsm_concat(fsm_symbol($2),fsm_concat(fsm_universal(),fsm_concat(fsm_symbol($2),fsm_concat(fsm_ignore($4,union_quantifiers(),OP_IGNORE_ALL),fsm_universal()))))); }
+| SUCCESSOR_OF network COMMA VAR RPAREN {$$ = fsm_concat(fsm_universal(),fsm_concat(fsm_ignore($2,union_quantifiers(),OP_IGNORE_ALL),fsm_concat(fsm_symbol($4),fsm_concat(fsm_universal(),fsm_concat(fsm_symbol($4),fsm_universal()))))); }
+| sub1 sub2 {$$ = fsm_substitute_symbol($1,subval1,subval2); substituting = 0; xxfree(subval1); xxfree(subval2); subval1 = subval2 = NULL;}
+
+sub1: SUBSTITUTE LBRACKET network COMMA { $$ = $3; substituting = 1; }
+sub2: SUBVAL COMMA SUBVAL RBRACKET { subval1 = $2; subval2 = $4; }
+
+network12: fend { $$ = $1; } |
+ ISIDENTITY network RPAREN { $$ = fsm_boolean(fsm_isidentity($2)); } |
+ ISFUNCTIONAL network RPAREN { $$ = fsm_boolean(fsm_isfunctional($2)); } |
+ ISUNAMBIGUOUS network RPAREN { $$ = fsm_boolean(fsm_isunambiguous($2)); } |
+ NOTID network RPAREN { $$ = fsm_extract_nonidentity(fsm_copy($2)); } |
+ LOWERUNIQ network RPAREN { $$ = fsm_lowerdet(fsm_copy($2)); } |
+ LOWERUNIQEPS network RPAREN { $$ = fsm_lowerdeteps(fsm_copy($2)); } |
+ ALLFINAL network RPAREN { $$ = fsm_markallfinal(fsm_copy($2)); } |
+ UNAMBIGUOUSPART network RPAREN { $$ = fsm_extract_unambiguous(fsm_copy($2)); } |
+ AMBIGUOUSPART network RPAREN { $$ = fsm_extract_ambiguous(fsm_copy($2)); } |
+ AMBIGUOUSDOMAIN network RPAREN { $$ = fsm_extract_ambiguous_domain(fsm_copy($2)); } |
+ LETTERMACHINE network RPAREN { $$ = fsm_letter_machine(fsm_copy($2)); } |
+ MARKFSMTAIL network COMMA network RPAREN { $$ = fsm_mark_fsm_tail($2,$4); } |
+ MARKFSMTAILLOOP network COMMA network RPAREN { $$ = fsm_add_loop($2,$4,1); } |
+ MARKFSMMIDLOOP network COMMA network RPAREN { $$ = fsm_add_loop($2,$4,0); } |
+ MARKFSMLOOP network COMMA network RPAREN { $$ = fsm_add_loop($2,$4,2); } |
+ ADDSINK network RPAREN { $$ = fsm_add_sink($2,1); } |
+ LEFTREWR network COMMA network RPAREN { $$ = fsm_left_rewr($2,$4); } |
+ FLATTEN network COMMA network RPAREN { $$ = fsm_flatten($2,$4); } |
+ SUBLABEL network COMMA network COMMA network RPAREN { $$ = fsm_substitute_label($2, fsm_network_to_char($4), $6); } |
+ CLOSESIGMA network RPAREN { $$ = fsm_close_sigma(fsm_copy($2), 0); } |
+ CLOSESIGMAUNK network RPAREN { $$ = fsm_close_sigma(fsm_copy($2), 1); } |
+ EQSUBSTRINGS network COMMA network COMMA network RPAREN { $$ = fsm_equal_substrings($2,$4,$6); }
+
+fstart: FUNCTION network COMMA
+{ frec++; fargptr[frec] = 0 ;declare_function_name($1) ; add_function_argument($2); }
+| FUNCTION network
+{ frec++; fargptr[frec] = 0 ;declare_function_name($1) ; add_function_argument($2); }
+
+fmid: fstart network COMMA { add_function_argument($2); }
+| fmid network COMMA { add_function_argument($2); }
+
+fend: fmid network RPAREN
+{ add_function_argument($2); if (($$ = function_apply(defined_nets, defined_funcs)) == NULL) YYERROR; }
+| fstart network RPAREN
+{ add_function_argument($2); if (($$ = function_apply(defined_nets, defined_funcs)) == NULL) YYERROR; }
+| fstart RPAREN
+{ if (($$ = function_apply(defined_nets, defined_funcs)) == NULL) YYERROR;}
+
+%%
diff --git a/reverse.c b/reverse.c
new file mode 100644
index 0000000..b03edb7
--- /dev/null
+++ b/reverse.c
@@ -0,0 +1,48 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include "foma.h"
+
+struct fsm *fsm_reverse(struct fsm *net) {
+ struct fsm *revnet;
+ struct fsm_construct_handle *revh;
+ struct fsm_read_handle *inh;
+ int i;
+
+ inh = fsm_read_init(net);
+ revh = fsm_construct_init(net->name);
+ fsm_construct_copy_sigma(revh, net->sigma);
+
+ while (fsm_get_next_arc(inh)) {
+ fsm_construct_add_arc_nums(revh, fsm_get_arc_target(inh)+1, fsm_get_arc_source(inh)+1, fsm_get_arc_num_in(inh), fsm_get_arc_num_out(inh));
+ }
+
+ while ((i = fsm_get_next_final(inh)) != -1) {
+ fsm_construct_add_arc_nums(revh, 0, i+1, EPSILON, EPSILON);
+ }
+ while ((i = fsm_get_next_initial(inh)) != -1) {
+ fsm_construct_set_final(revh, i+1);
+ }
+ fsm_construct_set_initial(revh, 0);
+ fsm_read_done(inh);
+ revnet = fsm_construct_done(revh);
+ revnet->is_deterministic = 0;
+ revnet->is_epsilon_free = 0;
+ fsm_destroy(net);
+ return(revnet);
+}
diff --git a/rewrite.c b/rewrite.c
new file mode 100644
index 0000000..3ab5a1e
--- /dev/null
+++ b/rewrite.c
@@ -0,0 +1,610 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include "foma.h"
+
+// Lower(X) puts X on output tape (may also be represented by @ID@ on input tape)
+// Upper(X) puts X on input tape
+// Unrewritten(X) X on input tape, not rewritten (aligned with @O@ symbols)
+// NotContain(X) MT does not contain MT configuration X
+
+// Boundary: every MT word begins and ends with boundary, i.e. the @#@ symbol on the input tape, output tape, and relevant semantic symbols
+
+/*
+
+ [ @O@ ] [ @I[@ ] [ @I@ ] [ @I]@ ] [ @I[]@ ]
+ [ @0@ ] [ @#0001@ ] [ @#0001@ ] [ @#0001@ ] [ @#0001@ ]
+ [ @#@ ] [ ANY|@0@ ] [ ANY|@0@ ] [ ANY|@0@ ] [ ANY|@0@ ]
+ [ @ID@ ] [ ANY|@ID|@0@ ] [ ANY|@ID|@0@ ] [ ANY|@ID|@0@ ] [ ANY|@ID|@0@ ]
+
+
+*/
+/* Special symbols used:
+ @0@ Epsilon
+ @O@ Outside rewrite
+ @I@ Inside rewrite
+ @I[@ Beginning of rewrite
+ @I[]@ Beginning and end of rewrite
+ @I]@ End of rewrite
+ @ID@ Identity symbol (= repeat symbol on previous tape at this position)
+ @#@ Boundary (the symbol .#. is mapped to this in contexts before the compilation procedure)
+ @#X@ X = rule number (one for each rule, starting with @#0001@)
+*/
+
+struct rewrite_batch {
+
+ struct rewrite_set *rewrite_set;
+ struct fsm *Rulenames;
+ struct fsm *ISyms;
+ struct fsm *ANY;
+ struct fsm *IOpen;
+ struct fsm *IClose;
+ struct fsm *ITape;
+ struct fsm *Any4Tape;
+ struct fsm *Epextend;
+ int num_rules;
+ char (*namestrings)[8];
+
+};
+
+char *specialsymbols[] = {"@0@","@O@","@I@","@I[@","@I[]@","@I]@","@ID@","@#@", NULL};
+
+void rewrite_add_special_syms(struct rewrite_batch *rb, struct fsm *net);
+struct fsm *rewrite_upper(struct rewrite_batch *rb, struct fsm *upper);
+struct fsm *rewrite_lower(struct rewrite_batch *rb, struct fsm *lower);
+struct fsm *rewrite_two_level(struct rewrite_batch *rb, struct fsm *lang, int rightside);
+struct fsm *rewr_context_restrict(struct rewrite_batch *rb, struct fsm *X, struct fsmcontexts *LR);
+struct fsm *rewr_contains(struct rewrite_batch *rb, struct fsm *lang);
+struct fsm *rewr_unrewritten(struct rewrite_batch *rb, struct fsm *lang);
+struct fsm *rewr_notleftmost(struct rewrite_batch *rb, struct fsm *lang, int rule_number, int arrow_type);
+struct fsm *rewr_notshortest(struct rewrite_batch *rb, struct fsm *lang, int rule_number);
+struct fsm *rewr_notlongest(struct rewrite_batch *rb, struct fsm *lang, int rule_number, int arrow_type);
+struct fsm *rewrite_tape_m_to_n_of_k(struct fsm *lang, int m, int n, int k);
+struct fsm *rewrite_cp(struct rewrite_batch *rb, struct fsm *upper, struct fsm *lower, int rule_number);
+struct fsm *rewrite_cp_transducer(struct rewrite_batch *rb, struct fsm *t, int rule_number);
+struct fsm *rewrite_cp_markup(struct rewrite_batch *rb, struct fsm *upper, struct fsm *lower1, struct fsm *lower2, int rule_number);
+struct fsm *rewrite_epextend(struct rewrite_batch *rb);
+struct fsm *rewrite_any_4tape(struct rewrite_batch *rb);
+struct fsm *rewrite_itape(struct rewrite_batch *rb);
+void rewrite_cleanup(struct rewrite_batch *rb);
+
+
+struct fsm *fsm_rewrite(struct rewrite_set *all_rules) {
+ struct rewrite_batch *rb;
+ struct rewrite_set *ruleset;
+ struct fsmrules *rules;
+ struct fsmcontexts *contexts;
+ struct fsm *RuleCP, *Base, *Boundary, *Outside, *CP, *C, *LeftExtend, *RightExtend, *Center;
+ int i, num_rules, rule_number, dir;
+ /* Count parallel rules */
+ for (ruleset = all_rules, num_rules = 0; ruleset != NULL; ruleset = ruleset->next) {
+ for (rules = ruleset->rewrite_rules; rules != NULL; rules = rules->next) {
+ num_rules++;
+ }
+ }
+
+ rb = xxcalloc(1, sizeof(struct rewrite_batch));
+ rb->rewrite_set = all_rules;
+ rb->num_rules = num_rules;
+ rb->namestrings = xxmalloc(sizeof rb->namestrings * num_rules);
+ for (i = 0; i < rb->num_rules; i++) {
+ sprintf(rb->namestrings[i], "@#%04i@", i+1);
+ }
+
+ rb->ISyms = fsm_minimize(fsm_union(fsm_symbol("@I@"), fsm_union(fsm_symbol("@I[]@"), fsm_union(fsm_symbol("@I[@"), fsm_symbol("@I]@")))));
+ rb->Rulenames = fsm_empty_set();
+ for (i = 1; i <= num_rules; i++) {
+ rb->Rulenames = fsm_minimize(fsm_union(rb->Rulenames, fsm_symbol(rb->namestrings[i-1])));
+ }
+ rb->ANY = fsm_identity();
+ rewrite_add_special_syms(rb, rb->ANY);
+
+ /* Add auxiliary symbols to all alphabets */
+ for (ruleset = all_rules; ruleset != NULL; ruleset = ruleset->next) {
+ for (rules = ruleset->rewrite_rules; rules != NULL; rules = rules->next) {
+ rewrite_add_special_syms(rb, rules->left);
+ rewrite_add_special_syms(rb, rules->right);
+ rewrite_add_special_syms(rb, rules->right2);
+ }
+ for (contexts = ruleset->rewrite_contexts; contexts != NULL; contexts = contexts->next) {
+ rewrite_add_special_syms(rb, contexts->left);
+ rewrite_add_special_syms(rb, contexts->right);
+ }
+ }
+ /* Get cross-product of every rule, according to its type */
+ RuleCP = fsm_empty_set();
+ for (ruleset = all_rules, rule_number = 1; ruleset != NULL; ruleset = ruleset->next) {
+ dir = ruleset->rule_direction;
+ for (rules = ruleset->rewrite_rules; rules != NULL; rules = rules->next) {
+ if (rules->right == NULL) {
+ /* T(x)-type rule */
+ CP = rewrite_cp_transducer(rb, fsm_copy(rules->left), rule_number);
+ rules->cross_product = fsm_copy(CP);
+ rules->right = fsm_minimize(fsm_lower(fsm_copy(rules->left)));
+ rules->left = fsm_minimize(fsm_upper(fsm_copy(rules->left)));
+ rewrite_add_special_syms(rb, rules->right);
+ rewrite_add_special_syms(rb, rules->left);
+ } else if (rules->right2 == NULL) {
+ /* Regular rewrite rule */
+ CP = rewrite_cp(rb, fsm_copy(rules->left), fsm_copy(rules->right), rule_number);
+ rules->cross_product = fsm_copy(CP);
+ } else {
+ /* A -> B ... C -type rule */
+ CP = rewrite_cp_markup(rb, fsm_copy(rules->left), fsm_copy(rules->right), fsm_copy(rules->right2), rule_number);
+ rules->cross_product = fsm_copy(CP);
+ }
+ RuleCP = fsm_minimize(fsm_union(RuleCP, CP));
+ rule_number++;
+ }
+ }
+
+ /* Create Base language */
+ Boundary = fsm_parse_regex("\"@O@\" \"@0@\" \"@#@\" \"@ID@\"", NULL, NULL);
+ Outside = fsm_minimize(fsm_concat(fsm_symbol("@O@"), fsm_concat(fsm_symbol("@0@"), fsm_concat(fsm_copy(rb->ANY), fsm_symbol("@ID@")))));
+ Base = fsm_minimize(fsm_concat(fsm_copy(Boundary), fsm_concat(fsm_kleene_star(fsm_union(RuleCP, Outside)), fsm_copy(Boundary))));
+ fsm_destroy(Boundary);
+ for (ruleset = all_rules, rule_number = 1; ruleset != NULL; ruleset = ruleset->next) {
+ dir = ruleset->rule_direction;
+ /* Replace all context spec with Upper/Lower, depending on rule_direction */
+ for (contexts = ruleset->rewrite_contexts; contexts != NULL; contexts = contexts->next) {
+ switch(dir) {
+ case OP_UPWARD_REPLACE:
+ contexts->cpleft = rewrite_upper(rb, fsm_copy(contexts->left));
+ contexts->cpright = rewrite_upper(rb, fsm_copy(contexts->right));
+ break;
+ case OP_RIGHTWARD_REPLACE:
+ contexts->cpleft = rewrite_lower(rb, fsm_copy(contexts->left));
+ contexts->cpright = rewrite_upper(rb, fsm_copy(contexts->right));
+ break;
+ case OP_LEFTWARD_REPLACE:
+ contexts->cpleft = rewrite_upper(rb, fsm_copy(contexts->left));
+ contexts->cpright = rewrite_lower(rb, fsm_copy(contexts->right));
+ break;
+ case OP_DOWNWARD_REPLACE:
+ contexts->cpleft = rewrite_lower(rb, fsm_copy(contexts->left));
+ contexts->cpright = rewrite_lower(rb, fsm_copy(contexts->right));
+ break;
+ case OP_TWO_LEVEL_REPLACE:
+ contexts->cpleft = rewrite_two_level(rb, fsm_copy(contexts->left), 0);
+ contexts->cpright = rewrite_two_level(rb, fsm_copy(contexts->right), 1);
+ break;
+ }
+ }
+ for (rules = ruleset->rewrite_rules; rules != NULL; rules = rules->next) {
+ /* Just the rule center w/ number without CP() contests */
+ /* Actually, maybe better to include CP(U,L) in this, very slow with e.g. a -> a || _ b^15 */
+ if (rules->arrow_type & ARROW_DOTTED) {
+ /* define EP Tape1of4("@O@") | [ Tape1of4("@I[@" "@I@"* "@I]@" | "@I[]@") & Tape3of4(~["@0@"*]) ] ; */
+ /* Additional constraint: 0->x is only allowed between EP _ EP */
+ /* The left and right sides can be checked separately */
+ /* ~[?* Center ~[EP ?*]] & ~[~[?* EP] Center ?*] */
+ Center = fsm_copy(rules->cross_product);
+ Base = fsm_intersect(fsm_intersect(Base, fsm_complement(fsm_concat(rewrite_any_4tape(rb), fsm_concat(fsm_copy(Center), fsm_complement(fsm_concat(rewrite_epextend(rb), rewrite_any_4tape(rb))))))), fsm_complement(fsm_concat(fsm_complement(fsm_concat(rewrite_any_4tape(rb), rewrite_epextend(rb))), fsm_concat(fsm_copy(Center), rewrite_any_4tape(rb)))));
+ }
+ if (ruleset->rewrite_contexts) {
+ Base = fsm_intersect(Base, rewr_context_restrict(rb, rules->cross_product, ruleset->rewrite_contexts));
+ }
+ /* Determine C (based on rule type) */
+ C = fsm_empty_set();
+ if ((rules->arrow_type & ARROW_RIGHT) && !(rules->arrow_type & ARROW_OPTIONAL)) {
+ C = fsm_union(C, rewr_unrewritten(rb, fsm_minimize(fsm_minus(fsm_copy(rules->left), fsm_empty_string()))));
+ }
+ if ((rules->arrow_type & ARROW_LEFT) && !(rules->arrow_type & ARROW_OPTIONAL)) {
+ C = fsm_union(C, rewr_unrewritten(rb, fsm_minimize(fsm_minus(fsm_copy(rules->right), fsm_empty_string()))));
+ }
+ if (rules->arrow_type & ARROW_LONGEST_MATCH) {
+ if (rules->arrow_type & ARROW_RIGHT) {
+ C = fsm_union(C, rewr_notleftmost(rb, rewrite_upper(rb, fsm_copy(rules->left)), rule_number, rules->arrow_type));
+ C = fsm_union(C, rewr_notlongest(rb, rewrite_upper(rb, fsm_copy(rules->left)), rule_number, rules->arrow_type));
+ }
+ if (rules->arrow_type & ARROW_LEFT) {
+ C = fsm_union(C, rewr_notleftmost(rb, rewrite_lower(rb, fsm_copy(rules->right)), rule_number, rules->arrow_type));
+ C = fsm_union(C, rewr_notlongest(rb, rewrite_lower(rb, fsm_copy(rules->right)), rule_number, rules->arrow_type));
+ }
+ }
+ if (rules->arrow_type & ARROW_SHORTEST_MATCH) {
+ if (rules->arrow_type & ARROW_RIGHT) {
+ C = fsm_union(C, rewr_notleftmost(rb, rewrite_upper(rb, fsm_copy(rules->left)), rule_number, rules->arrow_type));
+ C = fsm_union(C, rewr_notshortest(rb, rewrite_upper(rb, fsm_copy(rules->left)), rule_number));
+ }
+ if (rules->arrow_type & ARROW_LEFT) {
+ C = fsm_union(C, rewr_notleftmost(rb, rewrite_lower(rb, fsm_copy(rules->right)), rule_number, rules->arrow_type));
+ C = fsm_union(C, rewr_notshortest(rb, rewrite_lower(rb, fsm_copy(rules->right)), rule_number));
+ }
+ }
+ if (!ruleset->rewrite_contexts) {
+ if (rules->arrow_type & ARROW_DOTTED) {
+ Base = fsm_minus(Base, rewr_contains(rb, fsm_concat(rewrite_epextend(rb), rewrite_epextend(rb))));
+ } else {
+ Base = fsm_minus(Base, rewr_contains(rb, fsm_copy(C)));
+ }
+ }
+ for (contexts = ruleset->rewrite_contexts; contexts != NULL; contexts = contexts->next) {
+ /* Constraints: running intersect w/ Base */
+ /* NotContain(LC [Unrewritten|LM|...] RC) */
+ if (rules->arrow_type & ARROW_DOTTED) {
+ /* Extend left and right */
+ LeftExtend = fsm_minimize(fsm_intersect(fsm_concat(rewrite_any_4tape(rb), fsm_copy(contexts->cpleft)), fsm_concat(rewrite_any_4tape(rb), rewrite_epextend(rb))));
+ RightExtend = fsm_minimize(fsm_intersect(fsm_concat(rewrite_epextend(rb), rewrite_any_4tape(rb)), fsm_concat(fsm_copy(contexts->cpright), rewrite_any_4tape(rb))));
+ Base = fsm_minus(Base, rewr_contains(rb, fsm_minimize(fsm_concat(LeftExtend, RightExtend))));
+ } else {
+ Base = fsm_minus(Base, rewr_contains(rb, fsm_concat(fsm_copy(contexts->cpleft), fsm_concat(fsm_copy(C), fsm_copy(contexts->cpright)))));
+ }
+ }
+ rule_number++;
+ fsm_destroy(C);
+ }
+ }
+ Base = fsm_minimize(fsm_lower(fsm_compose(Base, fsm_parse_regex("[?:0]^4 [?:0 ?:0 ? ?]* [?:0]^4", NULL, NULL))));
+ Base = fsm_unflatten(Base, "@0@", "@ID@");
+
+ for (i = 0; specialsymbols[i] != NULL; i++) {
+ Base->sigma = sigma_remove(specialsymbols[i], Base->sigma);
+ }
+ for (rule_number = 1; rule_number <= num_rules; rule_number++)
+ Base->sigma = sigma_remove(rb->namestrings[rule_number-1], Base->sigma);
+
+ fsm_compact(Base);
+ sigma_sort(Base);
+ rewrite_cleanup(rb);
+ return Base;
+}
+
+void rewrite_cleanup(struct rewrite_batch *rb) {
+
+ if (rb->Rulenames != NULL)
+ fsm_destroy(rb->Rulenames);
+ if (rb->ISyms != NULL)
+ fsm_destroy(rb->ISyms);
+ if (rb->ANY != NULL)
+ fsm_destroy(rb->ANY);
+ if (rb->IOpen != NULL)
+ fsm_destroy(rb->IOpen);
+ if (rb->IClose != NULL)
+ fsm_destroy(rb->IClose);
+ if (rb->ITape != NULL)
+ fsm_destroy(rb->ITape);
+ if (rb->Any4Tape != NULL)
+ fsm_destroy(rb->Any4Tape);
+ if (rb->Epextend != NULL)
+ fsm_destroy(rb->Epextend);
+ if (rb->namestrings != NULL)
+ xxfree(rb->namestrings);
+ xxfree(rb);
+ return;
+}
+
+
+struct fsm *rewr_notlongest(struct rewrite_batch *rb, struct fsm *lang, int rule_number, int arrow_type) {
+ /* define NotLongest(X) [Upper(X)/Lower(X) & Tape1of4(IOpen Tape1Sig* ["@O@" | IOpen] Tape1Sig*)] */
+ struct fsm *nl, *flt, *rulenum;
+ nl = fsm_parse_regex("[\"@I[@\"|\"@I[]@\"] [\"@I[@\"|\"@I[]@\"|\"@I]@\"|\"@I@\"|\"@O@\"]* [\"@O@\"|\"@I[@\"|\"@I[]@\"] [\"@I[@\"|\"@I[]@\"|\"@I]@\"|\"@I@\"|\"@O@\"]*", NULL, NULL);
+ nl = rewrite_tape_m_to_n_of_k(nl, 1, 1, 4);
+ rulenum = fsm_minimize(fsm_concat(fsm_identity(), fsm_concat(fsm_symbol(rb->namestrings[rule_number-1]), fsm_concat(fsm_identity(), fsm_concat(fsm_identity(), fsm_universal())))));
+ nl = fsm_intersect(nl, rulenum);
+ /* lang can't end in @0@ */
+ if (arrow_type & ARROW_RIGHT) {
+ flt = fsm_parse_regex("[? ? ? ?]* [? ? [?-\"@0@\"] ?]", NULL, NULL);
+ } else {
+ flt = fsm_parse_regex("[? ? ? ?]* [? ? ? [?-\"@0@\"]]", NULL, NULL);
+ }
+ return fsm_minimize(fsm_intersect(fsm_intersect(nl, fsm_copy(lang)), flt));
+}
+
+struct fsm *rewr_notshortest(struct rewrite_batch *rb, struct fsm *lang, int rule_number) {
+ /* define NotShortest(X) [Upper/Lower(X) & Tape1of4("@I[@" \IClose*)] */
+ struct fsm *ns, *rulenum;
+ ns = fsm_parse_regex("[\"@I[@\"] \\[\"@I]@\"]*", NULL, NULL);
+ rulenum = fsm_minimize(fsm_concat(fsm_identity(), fsm_concat(fsm_symbol(rb->namestrings[rule_number-1]), fsm_concat(fsm_identity(), fsm_concat(fsm_identity(), fsm_universal())))));
+ ns = rewrite_tape_m_to_n_of_k(ns, 1, 1, 4);
+ ns = fsm_intersect(ns, rulenum);
+ return fsm_minimize(fsm_intersect(ns, fsm_copy(lang)));
+}
+
+struct fsm *rewr_notleftmost(struct rewrite_batch *rb, struct fsm *lang, int rule_number, int arrow_type) {
+ struct fsm *nl, *flt, *rulenum;
+ /* define Leftmost(X) [Upper/Lower(X) & Tape1of4("@O@" Tape1Sig* IOpen Tape1Sig*) ] */
+ nl = fsm_parse_regex("\"@O@\" [\"@O@\"]* [\"@I[@\"|\"@I[]@\"] [\"@I[@\"|\"@I[]@\"|\"@I]@\"|\"@I@\"|\"@O@\"]*", NULL, NULL);
+ nl = rewrite_tape_m_to_n_of_k(nl, 1, 1, 4);
+ rulenum = fsm_minimize(fsm_concat(fsm_concat(fsm_symbol("@O@"), fsm_concat(fsm_identity(), fsm_concat(fsm_identity(), fsm_identity()))), fsm_concat(fsm_kleene_star(fsm_concat(fsm_symbol("@O@"), fsm_concat(fsm_identity(), fsm_concat(fsm_identity(), fsm_identity())))), fsm_concat(fsm_union(fsm_symbol("@I[@"), fsm_symbol("@I[]@")), fsm_concat(fsm_symbol(rb->namestrings[rule_number-1]), fsm_universal())))));
+ nl = fsm_intersect(nl, rulenum);
+ if (arrow_type & ARROW_RIGHT) {
+ flt = fsm_parse_regex("[? ? ? ?]* [? ? [?-\"@0@\"] ?]", NULL, NULL);
+ } else {
+ flt = fsm_parse_regex("[? ? ? ?]* [? ? ? [?-\"@0@\"]]", NULL, NULL);
+ }
+ return fsm_minimize(fsm_intersect(fsm_intersect(nl, fsm_copy(lang)), flt));
+}
+
+struct fsm *rewr_unrewritten(struct rewrite_batch *rb, struct fsm *lang) {
+ /* define Unrewritten(X) [X .o. [0:"@O@" 0:"@0@" ? 0:"@ID@"]*].l; */
+ struct fsm *C;
+ C = fsm_minimize(fsm_kleene_star(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@O@")), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@0@")), fsm_concat(fsm_copy(rb->ANY), fsm_cross_product(fsm_empty_string(), fsm_symbol("@ID@")))))));
+ return fsm_minimize(fsm_lower(fsm_compose(lang, C)));
+}
+
+struct fsm *rewr_contains(struct rewrite_batch *rb, struct fsm *lang) {
+ /* define NotContain(X) ~[[Tape1Sig Tape2Sig Tape3Sig Tape4Sig]* X ?*]; */
+ return fsm_minimize(fsm_concat(rewrite_any_4tape(rb), fsm_concat(lang, rewrite_any_4tape(rb))));
+}
+
+struct fsm *rewrite_tape_m_to_n_of_k(struct fsm *lang, int m, int n, int k) {
+ /* [X .o. [0:?^(m-1) ?^(n-m+1) 0:?^(k-n)]*].l */
+ return fsm_minimize(fsm_lower(fsm_compose(lang, fsm_kleene_star(fsm_concat(fsm_concat_n(fsm_cross_product(fsm_empty_string(), fsm_identity()), m-1), fsm_concat(fsm_concat_n(fsm_identity(), n-m+1), fsm_concat_n(fsm_cross_product(fsm_empty_string(), fsm_identity()), k-n)))))));
+}
+
+struct fsm *rewrite_two_level(struct rewrite_batch *rb, struct fsm *lang, int rightside) {
+ struct fsm *Lower, *Upper, *Result;
+ Lower = rewrite_lower(rb, fsm_minimize(fsm_lower(fsm_copy(lang))));
+ Upper = rewrite_upper(rb, fsm_minimize(fsm_upper(lang)));
+ if (rightside == 1) {
+ Result = fsm_minimize(fsm_intersect(fsm_concat(Lower, rewrite_any_4tape(rb)), fsm_concat(Upper, rewrite_any_4tape(rb))));
+ } else {
+ Result = fsm_minimize(fsm_intersect(fsm_concat(rewrite_any_4tape(rb), Lower), fsm_concat(rewrite_any_4tape(rb), Upper)));
+ }
+ return Result;
+}
+
+struct fsm *rewrite_lower(struct rewrite_batch *rb, struct fsm *lower) {
+
+ /*
+ Lower:
+
+ [ @O@ | ISyms | ISyms ]*
+ [ @0@ | Rulenums | Rulenums ]
+ [ <R>,@#@ | @0@,R | R ]
+ [ @ID@ | <R> | @0@ ]
+
+ R = any real symbol
+ <R> = any real symbol, not inserted
+
+ */
+
+ struct fsm *One, *Two, *Three, *Filter;
+
+ One = fsm_minimize(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@O@")), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@0@")), fsm_concat(fsm_union(fsm_symbol("@#@"), fsm_copy(rb->ANY)), fsm_cross_product(fsm_empty_string(),fsm_symbol("@ID@"))))));
+
+ Two = fsm_minimize(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->ISyms)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->Rulenames)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_union(fsm_copy(rb->ANY), fsm_symbol("@0@"))), fsm_copy(rb->ANY)))));
+
+ Three = fsm_minimize(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->ISyms)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->Rulenames)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->ANY)), fsm_cross_product(fsm_empty_string(), fsm_symbol("@0@"))))));
+
+ Filter = fsm_minimize(fsm_kleene_star(fsm_union(One, fsm_union(Two, Three))));
+ return fsm_minimize(fsm_lower(fsm_compose(lower, Filter)));
+}
+
+struct fsm *rewrite_any_4tape(struct rewrite_batch *rb) {
+
+ /*
+ Upper:
+
+ [ @O@ | ISyms ]*
+ [ @0@ | Rulenums ]
+ [ <R>,@#@ | @0@,R ]
+ [ @ID@ | R, at ID@, at 0@ ]
+
+ R = any real symbol
+ <R> = any real symbol, not inserted
+ */
+ if (rb->Any4Tape == NULL) {
+ rb->Any4Tape = fsm_minimize(fsm_kleene_star(fsm_union(fsm_concat(fsm_symbol("@O@"), fsm_concat(fsm_symbol("@0@"), fsm_concat(fsm_union(fsm_copy(rb->ANY), fsm_symbol("@#@")), fsm_symbol("@ID@")))), fsm_concat(fsm_copy(rb->ISyms), fsm_concat(fsm_copy(rb->Rulenames), fsm_concat(fsm_union(fsm_copy(rb->ANY), fsm_symbol("@0@")), fsm_union(fsm_copy(rb->ANY), fsm_union(fsm_symbol("@ID@"), fsm_symbol("@0@")))))))));
+ }
+ return fsm_copy(rb->Any4Tape);
+}
+
+struct fsm *rewrite_upper(struct rewrite_batch *rb, struct fsm *upper) {
+ /*
+ Upper:
+
+ [ @O@ | ISyms | ISyms ]*
+ [ @0@ | Rulenums | Rulenums ]
+ [ <R>,@#@ | @0@ | <R> ]
+ [ @ID@ | R | R, at ID@, at 0@ ]
+
+ R = any real symbol
+ <R> = any real symbol, not inserted
+ */
+
+ struct fsm *One, *Two, *Three, *Filter;
+
+ One = fsm_minimize(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@O@")), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@0@")), fsm_concat(fsm_union(fsm_symbol("@#@"), fsm_copy(rb->ANY)), fsm_cross_product(fsm_empty_string(),fsm_symbol("@ID@"))))));
+
+ Two = fsm_minimize(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->ISyms)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->Rulenames)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_symbol("@0@")), fsm_cross_product(fsm_empty_string(), fsm_copy(rb->ANY))))));
+
+ Three = fsm_minimize(fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->ISyms)), fsm_concat(fsm_cross_product(fsm_empty_string(), fsm_copy(rb->Rulenames)), fsm_concat(fsm_copy(rb->ANY), fsm_cross_product(fsm_empty_string(), fsm_union(fsm_union(fsm_symbol("@0@"), fsm_copy(rb->ANY)), fsm_symbol("@ID@")))))));
+
+ Filter = fsm_minimize(fsm_kleene_star(fsm_union(One, fsm_union(Two, Three))));
+ return fsm_minimize(fsm_lower(fsm_compose(upper, Filter)));
+}
+
+struct fsm *rewrite_align(struct fsm *upper, struct fsm *lower) {
+ struct fsm *first, *second, *third, *align, *align2;
+ /* `[[`[[Tape1of2(upper "@0@"*) & Tape2of2(lower "@0@"*) & ~[[? ?]* "@0@" "@0@" [? ?]*]], %@%_IDENTITY%_SYMBOL%_%@,%@UNK%@] .o. [? ?|"@UNK@" "@UNK@":"@ID@"]*].l, %@UNK%@,%@%_IDENTITY%_SYMBOL%_%@] */
+ first = fsm_minimize(rewrite_tape_m_to_n_of_k(fsm_concat(upper, fsm_kleene_star(fsm_symbol("@0@"))), 1, 1, 2));
+ second = fsm_minimize(rewrite_tape_m_to_n_of_k(fsm_concat(lower, fsm_kleene_star(fsm_symbol("@0@"))), 2, 2, 2));
+ third = fsm_minimize(fsm_parse_regex("~[[? ?]* \"@0@\" \"@0@\" [? ?]*]", NULL, NULL));
+
+ align = fsm_minimize(fsm_intersect(third, fsm_intersect(first, second)));
+ align = fsm_minimize(fsm_substitute_symbol(align, "@_IDENTITY_SYMBOL_@", "@UNK@"));
+ align2 = fsm_minimize(fsm_lower(fsm_compose(align, fsm_parse_regex("[? ? | \"@UNK@\" \"@UNK@\":\"@ID@\" ]*", NULL, NULL))));
+ align2 = fsm_minimize(fsm_substitute_symbol(align2, "@UNK@", "@_IDENTITY_SYMBOL_@"));
+ return align2;
+}
+
+struct fsm *rewrite_align_markup(struct fsm *upper, struct fsm *lower1, struct fsm *lower2) {
+ struct fsm *first, *second, *third, *fourth, *fifth, *sixth, *align, *align2;
+ /* [Tape1of2("@0@"*) & Tape2of2(lower1)] [Tape1of2(upper) & Tape2of2("@ID@"*)] [ Tape1of2(lower1) & Tape2of2("@0@"*)] */
+ /* + make sure IDENTITY and UNKNOWN are taken care of */
+ first = fsm_minimize(rewrite_tape_m_to_n_of_k(fsm_kleene_star(fsm_symbol("@0@")), 1, 1, 2));
+ second = fsm_minimize(rewrite_tape_m_to_n_of_k(lower1, 2, 2, 2));
+ third = fsm_minimize(rewrite_tape_m_to_n_of_k(upper, 1, 1, 2));
+ fourth = fsm_minimize(rewrite_tape_m_to_n_of_k(fsm_kleene_star(fsm_symbol("@ID@")), 2, 2, 2));
+ fifth = fsm_minimize(rewrite_tape_m_to_n_of_k(fsm_kleene_star(fsm_symbol("@0@")), 1, 1, 2));
+ sixth = fsm_minimize(rewrite_tape_m_to_n_of_k(lower2, 2, 2, 2));
+ align = fsm_minimize(fsm_concat(fsm_intersect(first, second), fsm_concat(fsm_intersect(third, fourth), fsm_intersect(fifth, sixth))));
+ align = fsm_minimize(fsm_substitute_symbol(align, "@_IDENTITY_SYMBOL_@", "@UNK@"));
+ align2 = fsm_minimize(fsm_lower(fsm_compose(align, fsm_parse_regex("[? ? | \"@UNK@\" \"@UNK@\":\"@ID@\" ]*", NULL, NULL))));
+ align2 = fsm_minimize(fsm_substitute_symbol(align2, "@UNK@", "@_IDENTITY_SYMBOL_@"));
+ return align2;
+}
+
+struct fsm *rewrite_itape(struct rewrite_batch *rb) {
+ if (rb->ITape == NULL) {
+ rb->ITape = fsm_parse_regex("[\"@I[]@\" ? ? ? | \"@I[@\" ? ? ? [\"@I@\" ? ? ?]* \"@I]@\" ? [?-\"@0@\"] ? ] [\"@I]@\" ? \"@0@\" ?]* | 0" , NULL, NULL);
+ }
+ return fsm_copy(rb->ITape);
+}
+
+struct fsm *rewrite_cp_markup(struct rewrite_batch *rb, struct fsm *upper, struct fsm *lower1, struct fsm *lower2, int rule_number) {
+ /* Same as rewrite_cp, could be consolidated */
+ struct fsm *Aligned, *threetape, *rulenumtape;
+ /* define CP(X,Y) Tape23of3(Align2(X,Y)) & [ "@I[@" ? ? ["@I@" ? ?]* "@I]@" ? ? | "@I[]@" ? ? | 0 ] */
+ Aligned = rewrite_align_markup(upper, lower1, lower2);
+ Aligned = rewrite_tape_m_to_n_of_k(Aligned, 3, 4, 4);
+ threetape = fsm_minimize(fsm_intersect(Aligned, rewrite_itape(rb)));
+ rulenumtape = rewrite_tape_m_to_n_of_k(fsm_minimize(fsm_kleene_star(fsm_symbol(rb->namestrings[rule_number-1]))), 2, 2, 4);
+ return fsm_minimize(fsm_intersect(threetape, rulenumtape));
+}
+
+struct fsm *rewrite_cp_transducer(struct rewrite_batch *rb, struct fsm *t, int rule_number) {
+ struct fsm *Aligned, *threetape, *rulenumtape;
+ Aligned = fsm_flatten(t, fsm_symbol("@0@"));
+ Aligned = rewrite_tape_m_to_n_of_k(Aligned, 3, 4, 4);
+ threetape = fsm_minimize(fsm_intersect(Aligned, rewrite_itape(rb)));
+ rulenumtape = rewrite_tape_m_to_n_of_k(fsm_minimize(fsm_kleene_star(fsm_symbol(rb->namestrings[rule_number-1]))), 2, 2, 4);
+ return fsm_minimize(fsm_intersect(threetape, rulenumtape));
+}
+
+struct fsm *rewrite_cp(struct rewrite_batch *rb, struct fsm *upper, struct fsm *lower, int rule_number) {
+ struct fsm *Aligned, *threetape, *rulenumtape;
+ /* define CP(X,Y) Tape23of3(Align2(X,Y)) & [ "@I[@" ? ? ["@I@" ? ?]* "@I]@" ? ? | "@I[]@" ? ? | 0 ] */
+ Aligned = rewrite_align(upper, lower);
+ Aligned = rewrite_tape_m_to_n_of_k(Aligned, 3, 4, 4);
+ threetape = fsm_minimize(fsm_intersect(Aligned, rewrite_itape(rb)));
+ rulenumtape = rewrite_tape_m_to_n_of_k(fsm_minimize(fsm_kleene_star(fsm_symbol(rb->namestrings[rule_number-1]))), 2, 2, 4);
+ return fsm_minimize(fsm_intersect(threetape, rulenumtape));
+}
+
+void rewrite_add_special_syms(struct rewrite_batch *rb, struct fsm *net) {
+ int i;
+ if (net == NULL)
+ return;
+ sigma_substitute(".#.", "@#@", net->sigma); /* We convert boundaries to our interal rep. */
+ /* This is because sigma merging (fsm_merge_sigma()) is handled */
+ /* in a special way for .#., which we don't want here. */
+
+ for (i = 0; specialsymbols[i] != NULL; i++) {
+ if (sigma_find(specialsymbols[i], net->sigma) == -1)
+ sigma_add(specialsymbols[i], net->sigma);
+ }
+ for (i = 1; i <= rb->num_rules; i++) {
+ sigma_add(rb->namestrings[i-1], net->sigma);
+ }
+ sigma_sort(net);
+}
+
+
+void fsm_clear_contexts(struct fsmcontexts *contexts) {
+ struct fsmcontexts *c, *cp;
+ for (c = contexts; c != NULL; c = cp) {
+ fsm_destroy(c->left);
+ fsm_destroy(c->right);
+ fsm_destroy(c->cpleft);
+ fsm_destroy(c->cpright);
+ cp = c->next;
+ xxfree(c);
+ }
+}
+
+
+struct fsm *rewr_context_restrict(struct rewrite_batch *rb, struct fsm *X, struct fsmcontexts *LR) {
+
+ struct fsm *Var, *Notvar, *UnionL, *UnionP, *Result, *NewX, *Left, *Right;
+ struct fsmcontexts *pairs;
+
+ Var = fsm_symbol("@VARX@");
+ //Notvar = fsm_minimize(fsm_kleene_star(fsm_term_negation(fsm_symbol("@VARX@"))));
+ Notvar = fsm_minus(rewrite_any_4tape(rb), fsm_contains(fsm_symbol("@VARX@")));
+ /* We add the variable symbol to all alphabets to avoid ? matching it */
+ /* which would cause extra nondeterminism */
+
+ NewX = fsm_copy(X);
+ if (sigma_find("@VARX@", NewX->sigma) == -1) {
+ sigma_add("@VARX@", NewX->sigma);
+ sigma_sort(NewX);
+ }
+ UnionP = fsm_empty_set();
+
+ for (pairs = LR; pairs != NULL ; pairs = pairs->next) {
+ if (pairs->left == NULL) {
+ Left = fsm_empty_string();
+ } else {
+ Left = fsm_copy(pairs->cpleft);
+ sigma_add("@VARX@", Left->sigma);
+ sigma_sort(Left);
+ }
+ if (pairs->right == NULL) {
+ Right = fsm_empty_string();
+ } else {
+ Right = fsm_copy(pairs->cpright);
+ sigma_add("@VARX@", Right->sigma);
+ sigma_sort(Right);
+ }
+ UnionP = fsm_union(fsm_concat(Left, fsm_concat(fsm_copy(Var), fsm_concat(fsm_copy(Notvar), fsm_concat(fsm_copy(Var), Right)))), UnionP);
+ }
+ UnionL = fsm_concat(fsm_copy(Notvar), fsm_concat(fsm_copy(Var), fsm_concat(fsm_copy(NewX), fsm_concat(fsm_copy(Var), fsm_copy(Notvar)))));
+ Result = fsm_minus(UnionL, fsm_concat(fsm_copy(Notvar), fsm_concat(fsm_copy(UnionP), fsm_copy(Notvar))));
+
+ if (sigma_find("@VARX@", Result->sigma) != -1) {
+ Result = fsm_complement(fsm_substitute_symbol(Result, "@VARX@","@_EPSILON_SYMBOL_@"));
+ } else {
+ Result = fsm_complement(Result);
+ }
+ fsm_destroy(UnionP);
+ fsm_destroy(Var);
+ fsm_destroy(Notvar);
+ fsm_destroy(NewX);
+ return(Result);
+}
+
+struct fsm *rewrite_epextend(struct rewrite_batch *rb) {
+
+ struct fsm *one, *two, *allzeroupper, *threea, *threeb, *threec, *three;
+
+ /* 1. @O@ @0@ [ANY|@#@] @ID@ */
+ /* 2. @I[]@ @#Rule@ [ANY] [@ID@|@0@|ANY] */
+ /* 3a. @I[@ @#Rule@ [ANY] [@ID@|@0@|ANY] */
+ /* 3b. @I@ @#Rule@ [ANY] [@ID@|@0@|ANY] */
+ /* 3c. @I]@ @#Rule@ [ANY] [@ID@|@0@|ANY] */
+ /* 3. [3a|3b|3c] & ~[[? ? "@0@" ?]*] */
+
+ /* TODO lower version as well */
+
+ if (rb->Epextend == NULL) {
+ one = fsm_minimize(fsm_concat(fsm_symbol("@O@"), fsm_concat(fsm_symbol("@0@"), fsm_concat(fsm_union(fsm_copy(rb->ANY), fsm_symbol("@#@")), fsm_symbol("@ID@")))));
+ two = fsm_minimize(fsm_concat(fsm_symbol("@I[]@"), fsm_concat(fsm_copy(rb->Rulenames), fsm_concat(fsm_copy(rb->ANY), fsm_union(fsm_symbol("@0@"), fsm_union(fsm_symbol("@ID@"), fsm_copy(rb->ANY)))))));
+ allzeroupper = fsm_parse_regex("~[[? ? \"@0@\" ?]*]", NULL, NULL);
+ threea = fsm_minimize(fsm_concat(fsm_symbol("@I[@"), fsm_concat(fsm_copy(rb->Rulenames), fsm_concat(fsm_union(fsm_copy(rb->ANY), fsm_symbol("@0@")), fsm_union(fsm_symbol("@0@"), fsm_union(fsm_symbol("@ID@"), fsm_copy(rb->ANY)))))));
+ threeb = fsm_minimize(fsm_kleene_star(fsm_concat(fsm_symbol("@I@"), fsm_concat(fsm_copy(rb->Rulenames), fsm_concat(fsm_union(fsm_copy(rb->ANY), fsm_symbol("@0@")), fsm_union(fsm_symbol("@0@"), fsm_union(fsm_symbol("@ID@"), fsm_copy(rb->ANY))))))));
+ threec = fsm_minimize(fsm_concat(fsm_symbol("@I]@"), fsm_concat(fsm_copy(rb->Rulenames), fsm_concat(fsm_union(fsm_copy(rb->ANY), fsm_symbol("@0@")), fsm_union(fsm_symbol("@0@"), fsm_union(fsm_symbol("@ID@"), fsm_copy(rb->ANY)))))));
+ three = fsm_intersect(allzeroupper, fsm_concat(threea, fsm_concat(threeb, threec)));
+ rb->Epextend = fsm_minimize(fsm_union(fsm_union(one, two), three));
+ }
+ return fsm_copy(rb->Epextend);
+}
diff --git a/sigma.c b/sigma.c
new file mode 100644
index 0000000..afa6401
--- /dev/null
+++ b/sigma.c
@@ -0,0 +1,422 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <string.h>
+#include <stdlib.h>
+#include "foma.h"
+
+struct sigma *sigma_remove(char *symbol, struct sigma *sigma) {
+ struct sigma *sigma_start, *sigma_prev = NULL;
+ sigma_prev = NULL;
+ sigma_start = sigma;
+ for ( ; sigma != NULL && sigma->number != -1; sigma_prev = sigma, sigma=sigma->next) {
+ if (strcmp(sigma->symbol,symbol) == 0) {
+ if (sigma_prev == NULL) {
+ sigma_start = sigma->next;
+ xxfree(sigma->symbol);
+ xxfree(sigma);
+ } else {
+ (sigma_prev)->next = sigma->next;
+ xxfree(sigma->symbol);
+ xxfree(sigma);
+ }
+ break;
+ }
+ }
+ return(sigma_start);
+}
+
+struct sigma *sigma_remove_num(int num, struct sigma *sigma) {
+ struct sigma *sigma_start, *sigma_prev = NULL;
+ sigma_prev = NULL;
+ sigma_start = sigma;
+ for ( ; sigma != NULL && sigma->number != -1; sigma_prev = sigma, sigma=sigma->next) {
+ if (sigma->number == num) {
+ if (sigma_prev == NULL) {
+ sigma_start = sigma->next;
+ xxfree(sigma->symbol);
+ xxfree(sigma);
+ } else {
+ (sigma_prev)->next = sigma->next;
+ xxfree(sigma->symbol);
+ xxfree(sigma);
+ }
+ break;
+ }
+ }
+ return(sigma_start);
+}
+
+int sigma_add_special (int symbol, struct sigma *sigma) {
+ struct sigma *sigma_previous = NULL, *sigma_splice = NULL;
+ char *str = NULL;
+ if (symbol == EPSILON)
+ str = xxstrdup("@_EPSILON_SYMBOL_@");
+ if (symbol == IDENTITY)
+ str = xxstrdup("@_IDENTITY_SYMBOL_@");
+ if (symbol == UNKNOWN)
+ str = xxstrdup("@_UNKNOWN_SYMBOL_@");
+
+ /* Insert special symbols pre-sorted */
+ if (sigma->number == -1) {
+ sigma->number = symbol;
+ } else {
+ for (;(sigma != NULL) && (sigma->number < symbol) && (sigma->number!=-1); sigma_previous=sigma,sigma = sigma->next) {
+ }
+ sigma_splice = xxmalloc(sizeof(struct sigma));
+ if (sigma_previous != NULL) {
+ (sigma_previous)->next = sigma_splice;
+ sigma_splice->number = symbol;
+ sigma_splice->symbol = str;
+ (sigma_splice)->next = sigma;
+ return(symbol);
+ } else {
+ sigma_splice->symbol = sigma->symbol;
+ sigma_splice->number = sigma->number;
+ sigma_splice->next = sigma->next;
+ sigma->number = symbol;
+ sigma->symbol = str;
+ sigma->next = sigma_splice;
+ return(symbol);
+ }
+ }
+ sigma->next = NULL;
+ sigma->symbol = str;
+ return(symbol);
+}
+
+/* WARNING: this function will indeed add a symbol to sigma */
+/* but it's up to the user to sort the sigma (affecting arc numbers in the network) */
+/* before merge_sigma() is ever called */
+
+int sigma_add (char *symbol, struct sigma *sigma) {
+ int assert = -1;
+ struct sigma *sigma_previous = NULL, *sigma_splice = NULL;
+
+ /* Special characters */
+ if (strcmp(symbol, "@_EPSILON_SYMBOL_@") == 0)
+ assert = EPSILON;
+ if (strcmp(symbol,"@_IDENTITY_SYMBOL_@") == 0)
+ assert = IDENTITY;
+ if (strcmp(symbol,"@_UNKNOWN_SYMBOL_@") == 0)
+ assert = UNKNOWN;
+
+ /* Insert non-special in any order */
+ if (assert == -1) {
+ if (sigma->number == -1) {
+ sigma->number = 3;
+ } else {
+ for (; sigma->next != NULL; sigma = sigma->next) {
+ }
+ sigma->next = xxmalloc(sizeof(struct sigma));
+ if ((sigma->number)+1 < 3) {
+ (sigma->next)->number = 3;
+ } else {
+ (sigma->next)->number = (sigma->number)+1;
+ }
+ sigma = sigma->next;
+ }
+ sigma->next = NULL;
+ sigma->symbol = xxstrdup(symbol);
+ return(sigma->number);
+ } else {
+ /* Insert special symbols pre-sorted */
+ if (sigma->number == -1) {
+ sigma->number = assert;
+ } else {
+ for (;(sigma != NULL) && (sigma->number < assert) && (sigma->number!=-1); sigma_previous=sigma,sigma = sigma->next) {
+ }
+ sigma_splice = xxmalloc(sizeof(struct sigma));
+ if (sigma_previous != NULL) {
+ (sigma_previous)->next = sigma_splice;
+ sigma_splice->number = assert;
+ sigma_splice->symbol = xxmalloc(sizeof(char)*(strlen(symbol)+1));
+ strcpy(sigma_splice->symbol, symbol);
+ (sigma_splice)->next = sigma;
+ return(assert);
+ } else {
+ sigma_splice->symbol = sigma->symbol;
+ sigma_splice->number = sigma->number;
+ sigma_splice->next = sigma->next;
+ sigma->number = assert;
+ sigma->symbol = xxmalloc(sizeof(char)*(strlen(symbol)+1));
+ strcpy(sigma->symbol, symbol);
+ sigma->next = sigma_splice;
+ return(assert);
+ }
+ }
+ sigma->next = NULL;
+ sigma->symbol = xxstrdup(symbol);
+ return(assert);
+ }
+}
+
+/* Remove symbols that are never used from sigma and renumber */
+/* The variable force controls whether to remove even though */
+/* @ or ? is present */
+/* If force == 1, unused symbols are always removed regardless */
+
+void sigma_cleanup (struct fsm *net, int force) {
+ int i,j,first,maxsigma,*attested;
+ struct fsm_state *fsm;
+ struct sigma *sig, *sig_prev, *sign;
+
+ if (force == 0) {
+ if (sigma_find_number(IDENTITY, net->sigma) != -1)
+ return;
+ if (sigma_find_number(UNKNOWN, net->sigma) != -1)
+ return;
+ }
+
+ maxsigma = sigma_max(net->sigma);
+ if (maxsigma < 0) { return; }
+ attested = xxmalloc(sizeof(int)*(maxsigma+1));
+ for (i=0; i<=maxsigma; i++)
+ *(attested+i) = 0;
+ fsm = net->states;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->in >=0)
+ *(attested+(fsm+i)->in) = 1;
+ if ((fsm+i)->out >=0)
+ *(attested+(fsm+i)->out) = 1;
+ }
+ for (i=3,j=3; i<=maxsigma;i++ ) {
+ if (*(attested+i)) {
+ *(attested+i) = j;
+ j++;
+ }
+ }
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->in > 2)
+ (fsm+i)->in = *(attested+(fsm+i)->in);
+ if ((fsm+i)->out > 2)
+ (fsm+i)->out = *(attested+(fsm+i)->out);
+ }
+ sig_prev = NULL;
+ for (sig = net->sigma; sig != NULL && sig->number != -1; sig = sign) {
+ first = 1;
+ sign = sig->next;
+ if (!*(attested+(sig->number))) {
+ xxfree(sig->symbol);
+ xxfree(sig);
+ if (sig_prev != NULL) {
+ sig_prev->next = sign;
+ first = 0;
+ } else {
+ first = 0;
+ net->sigma = sign;
+ }
+ } else {
+ sig->number = sig->number >= 3 ? *(attested+(sig->number)) : sig->number;
+ }
+ if (first)
+ sig_prev = sig;
+ }
+ xxfree(attested);
+ return;
+}
+
+int sigma_max(struct sigma *sigma) {
+ int i;
+ if (sigma == NULL)
+ return -1;
+ for (i=-1; sigma != NULL; sigma = sigma->next)
+ i = sigma->number > i ? sigma->number : i;
+ return(i);
+}
+
+int sigma_size(struct sigma *sigma) {
+ int i;
+ for(i=0; sigma != NULL; sigma = sigma->next)
+ i++;
+ return(i);
+}
+
+struct fsm_sigma_list *sigma_to_list(struct sigma *sigma) {
+ struct fsm_sigma_list *sl;
+ struct sigma *s;
+ sl = xxcalloc(sigma_max(sigma)+1,sizeof(struct fsm_sigma_list));
+ for (s = sigma; s != NULL && s->number != -1; s = s->next) {
+ (sl+(s->number))->symbol = s->symbol;
+ }
+ return sl;
+}
+
+int sigma_add_number(struct sigma *sigma, char *symbol, int number) {
+ struct sigma *newsigma, *prev_sigma;
+ prev_sigma = NULL;
+ if (sigma->number == -1) {
+ sigma->symbol = xxstrdup(symbol);
+ sigma->number = number;
+ sigma->next = NULL;
+ return(1);
+ }
+ for (newsigma = sigma; newsigma != NULL; newsigma = newsigma->next) {
+ prev_sigma = newsigma;
+ }
+ newsigma = xxmalloc(sizeof(struct sigma));
+ newsigma->symbol = xxstrdup(symbol);
+ newsigma->number = number;
+ newsigma->next = NULL;
+ prev_sigma->next = newsigma;
+ return(1);
+}
+
+int sigma_find_number(int number, struct sigma *sigma) {
+ if (sigma == NULL)
+ return -1;
+ if (sigma->number == -1) {
+ return -1;
+ }
+ /* for (;(sigma != NULL) && (sigma->number <= number); sigma = sigma->next) { */
+ for (;(sigma != NULL) && (sigma->number != -1); sigma = sigma->next) {
+ if (number == sigma->number) {
+ return (sigma->number);
+ }
+ }
+ return -1;
+}
+char *sigma_string(int number, struct sigma *sigma) {
+ if (sigma == NULL)
+ return NULL;
+ if (sigma->number == -1) {
+ return NULL;
+ }
+ for (;(sigma != NULL) && (sigma->number != -1); sigma = sigma->next) {
+ if (number == sigma->number) {
+ return (sigma->symbol);
+ }
+ }
+ return NULL;
+}
+
+/* Substitutes string symbol for sub in sigma */
+/* no check for duplicates */
+int sigma_substitute(char *symbol, char *sub, struct sigma *sigma) {
+ if (sigma->number == -1) {
+ return -1;
+ }
+ for (; sigma != NULL && sigma->number != -1 ; sigma = sigma->next) {
+ if (strcmp(sigma->symbol, symbol) == 0) {
+ xxfree(sigma->symbol);
+ sigma->symbol = strdup(sub);
+ return(sigma->number);
+ }
+ }
+ return -1;
+}
+
+int sigma_find(char *symbol, struct sigma *sigma) {
+
+ if (sigma == NULL || sigma->number == -1) {
+ return -1;
+ }
+ for (; sigma != NULL && sigma->number != -1 ; sigma = sigma->next) {
+ if (strcmp(sigma->symbol, symbol) == 0) {
+ return (sigma->number);
+ }
+ }
+ return -1;
+}
+
+struct ssort {
+ char *symbol;
+ int number;
+};
+
+int ssortcmp(struct ssort *a, struct ssort *b) {
+ return(strcmp(a->symbol, b->symbol));
+}
+
+struct sigma *sigma_copy(struct sigma *sigma) {
+ int f = 0;
+ struct sigma *copy_sigma, *copy_sigma_s;
+
+ if (sigma == NULL) { return NULL; }
+ copy_sigma_s = xxmalloc(sizeof(struct sigma));
+
+ for (copy_sigma = copy_sigma_s; sigma != NULL; sigma=sigma->next) {
+ if (f == 1) {
+ copy_sigma->next = xxmalloc(sizeof(struct sigma));
+ copy_sigma = copy_sigma->next;
+ }
+ copy_sigma->number = sigma->number;
+ if (sigma->symbol != NULL)
+ copy_sigma->symbol = xxstrdup(sigma->symbol);
+ else
+ copy_sigma->symbol = NULL;
+ copy_sigma->next = NULL;
+ f = 1;
+ }
+ return(copy_sigma_s);
+}
+
+/* Assigns a consecutive numbering to symbols in sigma > IDENTITY */
+/* and sorts the sigma based on the symbol string contents */
+
+int sigma_sort(struct fsm *net) {
+ int(*comp)() = ssortcmp;
+ int size, i, max, *replacearray;
+ struct ssort *ssort;
+ struct sigma *sigma;
+ struct fsm_state *fsm_state;
+
+ size = sigma_max(net->sigma);
+ if (size < 0) { return 1; }
+ ssort = xxmalloc(sizeof(struct ssort)*size);
+
+ for (i=0, sigma=net->sigma; sigma != NULL; sigma=sigma->next) {
+ if (sigma->number > IDENTITY) {
+ ssort[i].symbol = (char *)sigma->symbol;
+ ssort[i].number = sigma->number;
+ i++;
+ }
+ }
+ max = i;
+ qsort(ssort, max, sizeof(struct ssort), comp);
+ replacearray = xxmalloc(sizeof(int)*(size+3));
+ for (i=0; i<max; i++)
+ replacearray[(ssort+i)->number] = i+3;
+
+ /* Replace arcs */
+ for(i=0, fsm_state = net->states; (fsm_state+i)->state_no != -1; i++) {
+ if ((fsm_state+i)->in > IDENTITY)
+ (fsm_state+i)->in = replacearray[(fsm_state+i)->in];
+ if ((fsm_state+i)->out > IDENTITY)
+ (fsm_state+i)->out = replacearray[(fsm_state+i)->out];
+ }
+ /* Replace sigma */
+ for (i=0, sigma=net->sigma; sigma != NULL; sigma=sigma->next) {
+ if (sigma->number > IDENTITY) {
+ sigma->number = i+3;
+ sigma->symbol = (ssort+i)->symbol;
+ i++;
+ }
+ }
+ xxfree(replacearray);
+ xxfree(ssort);
+ return(1);
+}
+
+struct sigma *sigma_create() {
+ struct sigma *sigma;
+ sigma = xxmalloc(sizeof(struct sigma));
+ sigma->number = -1; /*Empty sigma*/
+ sigma->next = NULL;
+ sigma->symbol = NULL;
+ return(sigma);
+}
diff --git a/spelling.c b/spelling.c
new file mode 100644
index 0000000..0d8cdff
--- /dev/null
+++ b/spelling.c
@@ -0,0 +1,880 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <limits.h>
+#include "foma.h"
+
+#define INITIAL_AGENDA_SIZE 256
+#define INITIAL_HEAP_SIZE 256
+#define INITIAL_STRING_SIZE 256
+#define MED_DEFAULT_LIMIT 4 /* Default max words to find */
+#define MED_DEFAULT_CUTOFF 15 /* Default MED cost cutoff */
+#define MED_DEFAULT_MAX_HEAP_SIZE 262145 /* By default won't grow heap more than this */
+
+#define BITMASK(b) (1 << ((b) & 7))
+#define BITSLOT(b) ((b) >> 3)
+#define BITSET(a,b) ((a)[BITSLOT(b)] |= BITMASK(b))
+#define BITCLEAR(a,b) ((a)[BITSLOT(b)] &= ~BITMASK(b))
+#define BITTEST(a,b) ((a)[BITSLOT(b)] & BITMASK(b))
+#define BITNSLOTS(nb) ((nb + CHAR_BIT - 1) / CHAR_BIT)
+#define min_(X, Y) ((X) < (Y) ? (X) : (Y))
+
+static int calculate_h(struct apply_med_handle *medh, int *intword, int currpos, int state);
+static struct astarnode *node_delete_min();
+int node_insert(struct apply_med_handle *medh, int wordpos, int fsmstate, int g, int h, int in, int out, int parent);
+
+char *print_sym(int sym, struct sigma *sigma) {
+ while (sigma != NULL) {
+ if (sigma->number == sym) {
+ return(sigma->symbol);
+ }
+ sigma = sigma->next;
+ }
+ return NULL;
+}
+
+void print_match(struct apply_med_handle *medh, struct astarnode *node, struct sigma *sigma, char *word) {
+ int sym, i, wordlen , printptr;
+ struct astarnode *n;
+ int_stack_clear();
+ wordlen = medh->wordlen;
+ for (n = node; n != NULL ; n = medh->agenda+(n->parent)) {
+ if (n->in == 0 && n->out == 0)
+ break;
+ if (n->parent == -1)
+ break;
+ int_stack_push(n->in);
+ }
+ printptr = 0;
+ if (medh->outstring_length < 2*wordlen) {
+ medh->outstring_length *= 2;
+ medh->outstring = xxrealloc(medh->outstring, medh->outstring_length*sizeof(char));
+ }
+ while (!(int_stack_isempty())) {
+ sym = int_stack_pop();
+ if (sym > 2) {
+ printptr += sprintf(medh->outstring+printptr,"%s", print_sym(sym, sigma));
+ }
+ if (sym == 0) {
+ if (medh->align_symbol) {
+ printptr += sprintf(medh->outstring+printptr,"%s",medh->align_symbol);
+ }
+ }
+ if (sym == 2) {
+ printptr += sprintf(medh->outstring+printptr,"@");
+ }
+ }
+ for (n = node; n != NULL ; n = medh->agenda+(n->parent)) {
+ if (n->in == 0 && n->out == 0)
+ break;
+ if (n->parent == -1)
+ break;
+ else
+ int_stack_push(n->out);
+ }
+ printptr = 0;
+ if (medh->instring_length < 2*wordlen) {
+ medh->instring_length *= 2;
+ medh->instring = xxrealloc(medh->instring, medh->instring_length*sizeof(char));
+ }
+ for (i = 0; !(int_stack_isempty()); ) {
+ sym = int_stack_pop();
+ if (sym > 2) {
+ printptr += sprintf(medh->instring+printptr,"%s", print_sym(sym, sigma));
+ i += utf8skip(word+i)+1;
+ }
+ if (sym == 0) {
+ if (medh->align_symbol) {
+ printptr += sprintf(medh->instring+printptr,"%s",medh->align_symbol);
+ }
+ }
+ if (sym == 2) {
+ if (i > wordlen) {
+ printptr += sprintf(medh->instring+printptr,"*");
+ } else {
+ //printf("%.*s", utf8skip(word+i)+1, word+i);
+ printptr += sprintf(medh->instring+printptr,"%.*s", utf8skip(word+i)+1, word+i);
+ i+= utf8skip(word+i)+1;
+ }
+ }
+ }
+ medh->cost = node->g;
+ // printf("Cost[f]: %i\n\n", node->g);
+}
+
+void apply_med_clear(struct apply_med_handle *medh) {
+ if (medh == NULL)
+ return;
+ if (medh->agenda != NULL)
+ xxfree(medh->agenda);
+ if (medh->instring != NULL)
+ xxfree(medh->instring);
+ if (medh->outstring != NULL)
+ xxfree(medh->outstring);
+ if (medh->heap != NULL)
+ xxfree(medh->heap);
+ if (medh->state_array != NULL)
+ xxfree(medh->state_array);
+ if (medh->align_symbol != NULL)
+ xxfree(medh->align_symbol);
+ if (medh->letterbits != NULL)
+ xxfree(medh->letterbits);
+ if (medh->nletterbits != NULL)
+ xxfree(medh->nletterbits);
+ if (medh->intword != NULL)
+ xxfree(medh->intword);
+ if (medh->sigmahash != NULL)
+ sh_done(medh->sigmahash);
+ xxfree(medh);
+}
+
+struct apply_med_handle *apply_med_init(struct fsm *net) {
+
+ struct apply_med_handle *medh;
+ struct sigma *sigma;
+ medh = xxcalloc(1,sizeof(struct apply_med_handle));
+ medh->net = net;
+ medh->agenda = xxmalloc(sizeof(struct astarnode)*INITIAL_AGENDA_SIZE);
+ medh->agenda->f = -1;
+ medh->agenda_size = INITIAL_AGENDA_SIZE;
+
+ medh->heap = xxmalloc(sizeof(int)*INITIAL_HEAP_SIZE);
+ medh->heap_size = INITIAL_HEAP_SIZE;
+ *(medh->heap) = 0; /* Points to sentinel */
+ medh->astarcount = 1;
+ medh->heapcount = 0;
+ medh->state_array = map_firstlines(net);
+ if (net->medlookup != NULL && net->medlookup->confusion_matrix != NULL) {
+ medh->hascm = 1;
+ medh->cm = net->medlookup->confusion_matrix;
+ }
+ medh->maxsigma = sigma_max(net->sigma)+1;
+ medh->sigmahash = sh_init();
+ for (sigma = net->sigma; sigma != NULL && sigma->number != -1 ; sigma=sigma->next ) {
+ if (sigma->number > IDENTITY) {
+ sh_add_string(medh->sigmahash, sigma->symbol, sigma->number);
+ }
+ }
+
+
+ fsm_create_letter_lookup(medh, net);
+
+ medh->instring = xxmalloc(sizeof(char)*INITIAL_STRING_SIZE);
+ medh->instring_length = INITIAL_STRING_SIZE;
+ medh->outstring = xxmalloc(sizeof(char)*INITIAL_STRING_SIZE);
+ medh->outstring_length = INITIAL_STRING_SIZE;
+
+ medh->med_limit = MED_DEFAULT_LIMIT;
+ medh->med_cutoff = MED_DEFAULT_CUTOFF;
+ medh->med_max_heap_size = MED_DEFAULT_MAX_HEAP_SIZE;
+ return(medh);
+}
+
+void apply_med_set_heap_max(struct apply_med_handle *medh, int max) {
+ if (medh != NULL) {
+ medh->med_max_heap_size = max;
+ }
+}
+
+void apply_med_set_align_symbol(struct apply_med_handle *medh, char *align) {
+ if (medh != NULL) {
+ medh->align_symbol = xxstrdup(align);
+ }
+}
+
+void apply_med_set_med_limit(struct apply_med_handle *medh, int max) {
+ if (medh != NULL) {
+ medh->med_limit = max;
+ }
+}
+
+void apply_med_set_med_cutoff(struct apply_med_handle *medh, int max) {
+ if (medh != NULL) {
+ medh->med_cutoff = max;
+ }
+}
+
+int apply_med_get_cost(struct apply_med_handle *medh) {
+ return(medh->cost);
+}
+
+char *apply_med_get_instring(struct apply_med_handle *medh) {
+ return(medh->instring);
+}
+
+char *apply_med_get_outstring(struct apply_med_handle *medh) {
+ return(medh->outstring);
+}
+
+char *apply_med(struct apply_med_handle *medh, char *word) {
+
+ /* local ok: i, j, target, in, out, g, h, curr_node */
+ /* not ok: curr_ptr, curr_pos, lines, nummatches, nodes_expanded, curr_state */
+
+ int i, j, target, in, out, g, h, thisskip;
+
+ int delcost, subscost, inscost;
+ char temputf[5] ;
+
+ struct astarnode *curr_node;
+
+ delcost = subscost = inscost = 1;
+
+
+ if (word == NULL) {
+ goto resume;
+ }
+
+ medh->word = word;
+
+ medh->nodes_expanded = 0;
+ medh->astarcount = 1;
+ medh->heapcount = 0;
+
+ medh->wordlen = strlen(word);
+ medh->utf8len = utf8strlen(word);
+ if (medh->intword != NULL) {
+ xxfree(medh->intword);
+ }
+ medh->intword = xxmalloc(sizeof(int)*(medh->utf8len+1));
+
+ /* intword -> sigma numbers of word */
+ for (i=0, j=0; i < medh->wordlen; i += thisskip, j++) {
+ thisskip = utf8skip(word+i)+1;
+ strncpy(temputf, word+i, thisskip);
+ temputf[thisskip] = '\0';
+ if (sh_find_string(medh->sigmahash, temputf) != NULL) {
+ *(medh->intword+j) = sh_get_value(medh->sigmahash);
+ } else {
+ *(medh->intword+j) = IDENTITY;
+ }
+ }
+
+
+
+ *(medh->intword+j) = -1; /* sentinel */
+
+ /* Insert (0,0) g = 0 */
+
+ h = calculate_h(medh, medh->intword, 0, 0);
+
+ /* Root node */
+
+ if (!node_insert(medh,0,0,0,h,0,0,-1)) { goto out; }
+ medh->nummatches = 0;
+
+ for(;;) {
+
+ curr_node = node_delete_min(medh);
+ /* Need to save this in case we realloc and print_match() */
+ medh->curr_agenda_offset = curr_node-medh->agenda;
+ if (curr_node == NULL) {
+ //printf("Reached cutoff of %i.\n", medh->med_cutoff);
+ goto out;
+ }
+ medh->curr_state = curr_node->fsmstate;
+ medh->curr_ptr = (medh->state_array+medh->curr_state)->transitions;
+ if (!medh->curr_ptr->final_state || !(curr_node->wordpos == medh->utf8len)) {
+ //continue;
+ }
+
+ medh->nodes_expanded++;
+
+ if (curr_node->f > medh->med_cutoff) {
+ //printf("Reached cutoff of %i\n", medh->med_cutoff);
+ goto out;
+ }
+
+ medh->curr_pos = curr_node->wordpos;
+ medh->curr_state = curr_node->fsmstate;
+ medh->curr_g = curr_node->g;
+
+ medh->lines = 0;
+ medh->curr_node_has_match = 0;
+
+ for (medh->curr_ptr = (medh->state_array+medh->curr_state)->transitions ; ;) {
+ if (medh->curr_ptr->state_no == -1) {
+ break;
+ }
+ medh->lines++;
+ if (medh->curr_ptr->final_state && medh->curr_pos == medh->utf8len) {
+ if (medh->curr_node_has_match == 0) {
+ /* Found a match */
+ medh->curr_node_has_match = 1;
+ print_match(medh, medh->agenda+medh->curr_agenda_offset, medh->net->sigma, medh->word);
+ medh->nummatches++;
+ return(medh->outstring);
+ }
+ }
+
+ resume:
+
+ if (medh->nummatches == medh->med_limit) {
+ goto out;
+ }
+
+ if (medh->curr_ptr->target == -1 && medh->curr_pos == medh->utf8len)
+ break;
+ if (medh->curr_ptr->target == -1 && medh->lines == 1)
+ goto insert;
+ if (medh->curr_ptr->target == -1)
+ break;
+
+ target = medh->curr_ptr->target;
+ /* Add nodes to edge:0, edge:input, 0:edge */
+
+ /* Delete a symbol from input */
+ in = medh->curr_ptr->in;
+ out = 0;
+ g = medh->hascm ? medh->curr_g + *(medh->cm+in*medh->maxsigma) : medh->curr_g + delcost;
+ h = calculate_h(medh, medh->intword, medh->curr_pos, medh->curr_ptr->target);
+
+ if ((medh->curr_pos == medh->utf8len) && (medh->curr_ptr->final_state == 0) && (h == 0)) {
+ // h = 1;
+ }
+
+ if (g+h <= medh->med_cutoff) {
+ if (!node_insert(medh, medh->curr_pos, target, g, h, in, out, medh->curr_agenda_offset)) {
+ goto out;
+ }
+ }
+ if (medh->curr_pos == medh->utf8len)
+ goto skip;
+
+ /* Match/substitute */
+ in = medh->curr_ptr->in;
+ out = *(medh->intword+medh->curr_pos);
+ if (in != out) {
+ g = medh->hascm ? medh->curr_g + *(medh->cm+in*medh->maxsigma+out) : medh->curr_g + subscost;
+ } else {
+ g = medh->curr_g;
+ }
+
+ h = calculate_h(medh, medh->intword, medh->curr_pos+1, medh->curr_ptr->target);
+ if ((g+h) <= medh->med_cutoff) {
+ if (!node_insert(medh,medh->curr_pos+1, target, g, h, in, out, medh->curr_agenda_offset)) {
+ goto out;
+ }
+ }
+ insert:
+ /* Insert a symbol into input */
+ /* Can only be done once per state */
+
+ if (medh->lines == 1) {
+
+ in = 0;
+ out = *(medh->intword+medh->curr_pos);
+
+ g = medh->hascm ? medh->curr_g + *(medh->cm+out) : medh->curr_g + inscost;
+ h = calculate_h(medh, medh->intword, medh->curr_pos+1, medh->curr_state);
+
+ if (g+h <= medh->med_cutoff)
+ if (!node_insert(medh,medh->curr_pos+1, medh->curr_state, g, h, in, out, medh->curr_agenda_offset)) {
+ goto out;
+ }
+ }
+ if (medh->curr_ptr->target == -1)
+ break;
+ skip:
+ if ((medh->curr_ptr+1)->state_no == (medh->curr_ptr)->state_no)
+ medh->curr_ptr++;
+ else
+ break;
+ }
+ }
+ out:
+ return(NULL);
+}
+
+int calculate_h(struct apply_med_handle *medh, int *intword, int currpos, int state) {
+ int i, j, hinf, hn, curr_sym;
+ uint8_t *bitptr, *nbitptr;
+ hinf = 0;
+ hn = 0;
+
+ bitptr = state*medh->bytes_per_letter_array + medh->letterbits;
+ nbitptr = state*medh->bytes_per_letter_array + medh->nletterbits;
+
+ /* For n = inf */
+ if (*(intword+currpos) == -1)
+ return 0;
+ for (i = currpos; *(intword+i) != -1; i++) {
+ curr_sym = *(intword+i);
+ if (!BITTEST(bitptr, curr_sym)) {
+ hinf++;
+ }
+ }
+ /* For n = maxdepth */
+ if (*(intword+currpos) == -1)
+ return 0;
+ for (i = currpos, j = 0; j < medh->maxdepth && *(intword+i) != -1; i++, j++) {
+ curr_sym = *(intword+i);
+ if (!BITTEST(nbitptr, curr_sym)) {
+ hn++;
+ }
+ }
+ return(hinf > hn ? hinf : hn);
+}
+
+struct astarnode *node_delete_min(struct apply_med_handle *medh) {
+ int i, child;
+ struct astarnode *firstptr, *lastptr;
+ if (medh->heapcount == 0) {
+ return NULL;
+ }
+
+ /* We find the min from the heap */
+
+ firstptr = medh->agenda+medh->heap[1];
+ lastptr = medh->agenda+medh->heap[medh->heapcount];
+ medh->heapcount--;
+
+ /* Adjust heap */
+ for (i = 1; (i<<1) <= medh->heapcount; i = child) {
+ child = i<<1;
+
+ /* If right child is smaller (higher priority) than left child */
+ if (child != medh->heapcount &&
+ ((medh->agenda+medh->heap[child+1])->f < (medh->agenda+medh->heap[child])->f ||
+ ((medh->agenda+medh->heap[child+1])->f <= (medh->agenda+medh->heap[child])->f &&
+ (medh->agenda+medh->heap[child+1])->wordpos > (medh->agenda+medh->heap[child])->wordpos))) {
+ child++;
+ }
+
+ /* If child has lower priority than last element */
+ if ((medh->agenda+medh->heap[child])->f < lastptr->f ||
+ ((medh->agenda+medh->heap[child])->f <= lastptr->f &&
+ (medh->agenda+medh->heap[child])->wordpos > lastptr->wordpos)) {
+
+ medh->heap[i] = medh->heap[child];
+ } else {
+ break;
+ }
+ }
+ medh->heap[i] = (lastptr-medh->agenda);
+ return(firstptr);
+}
+
+int node_insert(struct apply_med_handle *medh, int wordpos, int fsmstate, int g, int h, int in, int out, int parent) {
+ int i,j,f;
+ /* We add the node in the array */
+ i = medh->astarcount;
+ if (i >= (medh->agenda_size-1)) {
+ if (medh->agenda_size*2 >= medh->med_max_heap_size) {
+ //printf("heap limit reached by %i\n", medh->med_max_heap_size);
+ return 0;
+ }
+ medh->agenda_size *= 2;
+ medh->agenda = xxrealloc(medh->agenda, sizeof(struct astarnode)*medh->agenda_size);
+ }
+ f = g + h;
+ (medh->agenda+i)->wordpos = wordpos;
+ (medh->agenda+i)->fsmstate = fsmstate;
+ (medh->agenda+i)->f = f;
+ (medh->agenda+i)->g = g;
+ (medh->agenda+i)->h = h;
+ (medh->agenda+i)->in = in;
+ (medh->agenda+i)->out = out;
+ (medh->agenda+i)->parent = parent;
+ medh->astarcount++;
+
+ /* We also put the ptr on the heap */
+ medh->heapcount++;
+
+ if (medh->heapcount == medh->heap_size-1) {
+ medh->heap = xxrealloc(medh->heap, sizeof(int)*medh->heap_size*2);
+ medh->heap_size *= 2;
+ }
+ /* >= makes fifo */
+ // for (j = heapcount; (agenda+heap[j/2])->f > f; j /= 2) {
+ for (j = medh->heapcount; ( (medh->agenda+medh->heap[j>>1])->f > f) || ((medh->agenda+medh->heap[j>>1])->f >= f && (medh->agenda+medh->heap[j>>1])->wordpos <= wordpos) ; j = j >> 1) {
+ medh->heap[j] = medh->heap[j>>1];
+ }
+ medh->heap[j] = i;
+ return 1;
+}
+
+
+void letterbits_union(int v, int vp, uint8_t *ptr, int bytes_per_letter_array) {
+ int i;
+ uint8_t *vptr, *vpptr;
+ vptr = ptr+(v*bytes_per_letter_array);
+ vpptr = ptr+(vp*bytes_per_letter_array);
+ for (i=0; i < bytes_per_letter_array; i++) {
+ *(vptr+i) = *(vptr+i)|*(vpptr+i);
+ }
+}
+
+void letterbits_copy(int source, int target, uint8_t *ptr, int bytes_per_letter_array) {
+ int i;
+ uint8_t *sourceptr, *targetptr;
+ sourceptr = ptr+(source*bytes_per_letter_array);
+ targetptr = ptr+(target*bytes_per_letter_array);
+ for (i=0; i < bytes_per_letter_array; i++) {
+ *(targetptr+i) = *(sourceptr+i);
+ }
+}
+
+void letterbits_add(int v, int symbol, uint8_t *ptr, int bytes_per_letter_array) {
+ uint8_t *vptr;
+ vptr = ptr+(v*bytes_per_letter_array);
+ BITSET(vptr, symbol);
+}
+
+/* Creates, for each state, a list of symbols that can be matched */
+/* somewhere in subsequent paths (the case n = inf) */
+/* This is needed for h() of A*-search in approximate matching */
+/* This is done by a DFS where each state gets the properties */
+/* of the arcs it has as well as a copy of the properties of the target state */
+/* At the same time we keep track of the strongly connected components */
+/* And copy the properties from each root of the SCC to the children */
+/* The SCC part is required for the algorithm to work with cyclic graphs */
+/* This is basically a modification of Tarjan's (1972) SCC algorithm */
+/* However, it's converted from recursive form to iterative form using */
+/* goto statements. Here's the original recursive specification: */
+
+/* Input: FSM = (V,E) */
+/* Output: bitvector v.letters for each state */
+/* index = 1 DFS node number counter */
+
+/* procedure Mark(v) */
+/* v.index = index Set the depth index for v */
+/* v.lowlink = index */
+/* index++ */
+/* push(v) Push v on the stack */
+/* forall edge = (v, v') in E do Consider successors of v */
+/* v.letters |= v'.letters | edge Copy target v'.letters */
+/* if (v'.index == 0) Was successor v' visited? */
+/* Mark(v') Recurse */
+/* v.lowlink = min(v.lowlink, v'.lowlink) */
+/* elseif (v' is on stack) Was successor v' in stack S? */
+/* v.lowlink = min(v.lowlink, v'.index) */
+/* if (v.lowlink == v.index) Is v the root of a SCC? */
+/* do */
+/* pop(v') */
+/* v'.letters = v.letters */
+/* while (v' != v) */
+
+/* For keeping track of the strongly connected components */
+/* when doing the DFS */
+
+struct sccinfo {
+ int index;
+ int lowlink;
+ int on_t_stack;
+};
+
+void fsm_create_letter_lookup(struct apply_med_handle *medh, struct fsm *net) {
+
+ int num_states, num_symbols, index, v, vp, copystate, i, j;
+ struct fsm_state *curr_ptr;
+ struct sccinfo *sccinfo;
+ int depth;
+ medh->maxdepth = 2;
+
+ num_states = net->statecount;
+ num_symbols = sigma_max(net->sigma);
+
+ medh->bytes_per_letter_array = BITNSLOTS(num_symbols+1);
+ medh->letterbits = xxcalloc(medh->bytes_per_letter_array*num_states,sizeof(uint8_t));
+ medh->nletterbits = xxcalloc(medh->bytes_per_letter_array*num_states,sizeof(uint8_t));
+
+ sccinfo = xxcalloc(num_states,sizeof(struct sccinfo));
+
+ index = 1;
+ curr_ptr = net->states;
+ goto l1;
+
+ /* Here we go again, converting a recursive algorithm to an iterative one */
+ /* by gotos */
+
+ while(!ptr_stack_isempty()) {
+
+ curr_ptr = ptr_stack_pop();
+
+ v = curr_ptr->state_no; /* source state number */
+ vp = curr_ptr->target; /* target state number */
+
+ /* T: v.letterlist = list_union(v'->list, current edge label) */
+
+ letterbits_union(v, vp, medh->letterbits,medh->bytes_per_letter_array); /* add v' target bits to v */
+ letterbits_add(v, curr_ptr->in, medh->letterbits,medh->bytes_per_letter_array); /* add current arc label to v */
+
+ (sccinfo+v)->lowlink = min_((sccinfo+v)->lowlink,(sccinfo+vp)->lowlink);
+
+ if ((curr_ptr+1)->state_no != curr_ptr->state_no) {
+ goto l4;
+ } else {
+ goto l3;
+ }
+
+ l1:
+ v = curr_ptr->state_no;
+ vp = curr_ptr->target; /* target */
+ /* T: v.lowlink = index, index++, Tpush(v) */
+ (sccinfo+v)->index = index;
+ (sccinfo+v)->lowlink = index;
+ index++;
+ int_stack_push(v);
+ (sccinfo+v)->on_t_stack = 1;
+ /* if v' not visited (is v'.index set) */
+
+ /* If done here, check lowlink, pop */
+
+ if (vp == -1)
+ goto l4;
+ l2:
+ letterbits_add(v, curr_ptr->in, medh->letterbits,medh->bytes_per_letter_array);
+ if ((sccinfo+vp)->index == 0) {
+ /* push (v,e) ptr on stack */
+ ptr_stack_push(curr_ptr);
+ curr_ptr = (medh->state_array+(curr_ptr->target))->transitions;
+ /* (v,e) = (v',firstedge), goto init */
+ goto l1;
+ /* if v' visited */
+ /* T: v.lowlink = min(v.lowlink, v'.lowlink), union v.list with e, move to next edge in v, goto loop */
+ } else if ((sccinfo+vp)->on_t_stack) {
+ (sccinfo+v)->lowlink = min_((sccinfo+v)->lowlink,(sccinfo+vp)->lowlink);
+ }
+ /* If node is visited, copy its bits */
+ letterbits_union(v,vp,medh->letterbits,medh->bytes_per_letter_array);
+
+ l3:
+ if ((curr_ptr+1)->state_no == curr_ptr->state_no) {
+ curr_ptr++;
+ v = curr_ptr->state_no;
+ vp = curr_ptr->target; /* target */
+ goto l2;
+ }
+
+
+ /* T: if lastedge, v.lowlink == v.index, pop Tstack until v is popped and copy v.list to others */
+ /* Copy all bits from root of SCC to descendants */
+ l4:
+ if ((sccinfo+v)->lowlink == (sccinfo+v)->index) {
+ //printf("\nSCC: [%i] ",v);
+ while((copystate = int_stack_pop()) != v) {
+ (sccinfo+copystate)->on_t_stack = 0;
+ letterbits_copy(v, copystate, medh->letterbits, medh->bytes_per_letter_array);
+ //printf("%i ", copystate);
+ }
+ (sccinfo+v)->on_t_stack = 0;
+ //printf("\n");
+ }
+ }
+
+ for (i=0; i < num_states; i++) {
+ //printf("State %i: ",i);
+ for (j=0; j <= num_symbols; j++) {
+ if (BITTEST(medh->letterbits+(i*medh->bytes_per_letter_array),j)) {
+ //printf("[%i]",j);
+ }
+ }
+ //printf("\n");
+ }
+ int_stack_clear();
+
+ /* We do the same thing for some finite n (up to maxdepth) */
+ /* and store the result in nletterbits */
+
+ for (v=0; v < num_states; v++) {
+ ptr_stack_push((medh->state_array+v)->transitions);
+ int_stack_push(0);
+ while (!ptr_stack_isempty()) {
+ curr_ptr = ptr_stack_pop();
+ depth = int_stack_pop();
+ looper:
+ if (depth == medh->maxdepth)
+ continue;
+ if (curr_ptr->in != -1) {
+ letterbits_add(v, curr_ptr->in, medh->nletterbits, medh->bytes_per_letter_array);
+ }
+ if (curr_ptr->target != -1) {
+ if (curr_ptr->state_no == (curr_ptr+1)->state_no) {
+ ptr_stack_push(curr_ptr+1);
+ int_stack_push(depth);
+ }
+ depth++;
+ curr_ptr = (medh->state_array+(curr_ptr->target))->transitions;
+ goto looper;
+ }
+ }
+ }
+ for (i=0; i < num_states; i++) {
+ //printf("State %i: ",i);
+ for (j=0; j <= num_symbols; j++) {
+ if (BITTEST(medh->nletterbits+(i*medh->bytes_per_letter_array),j)) {
+ //printf("[%i]",j);
+ }
+ }
+ //printf("\n");
+ }
+ xxfree(sccinfo);
+}
+
+void cmatrix_print_att(struct fsm *net, FILE *outfile) {
+ int i, j, *cm, maxsigma;
+ maxsigma = sigma_max(net->sigma) + 1;
+ cm = net->medlookup->confusion_matrix;
+
+
+ for (i = 0; i < maxsigma ; i++) {
+ for (j = 0; j < maxsigma ; j++) {
+ if ((i != 0 && i < 3) || (j != 0 && j < 3)) { continue; }
+ if (i == 0 && j != 0) {
+ fprintf(outfile,"0\t0\t%s\t%s\t%i\n", "@0@", sigma_string(j, net->sigma), *(cm+i*maxsigma+j));
+ } else if (j == 0 && i != 0) {
+ fprintf(outfile,"0\t0\t%s\t%s\t%i\n", sigma_string(i,net->sigma), "@0@", *(cm+i*maxsigma+j));
+ } else if (j != 0 && i != 0) {
+ fprintf(outfile,"0\t0\t%s\t%s\t%i\n", sigma_string(i,net->sigma),sigma_string(j, net->sigma), *(cm+i*maxsigma+j));
+ }
+ }
+ }
+ fprintf(outfile,"0\n");
+}
+
+void cmatrix_print(struct fsm *net) {
+ int lsymbol, i, j, *cm, maxsigma;
+ char *thisstring;
+ struct sigma *sigma;
+ maxsigma = sigma_max(net->sigma) + 1;
+ cm = net->medlookup->confusion_matrix;
+
+ lsymbol = 0 ;
+ for (sigma = net->sigma; sigma != NULL; sigma = sigma->next) {
+ if (sigma->number < 3)
+ continue;
+ lsymbol = strlen(sigma->symbol) > lsymbol ? strlen(sigma->symbol) : lsymbol;
+ }
+ printf("%*s",lsymbol+2,"");
+ printf("%s","0 ");
+
+ for (i = 3; ; i++) {
+ if ((thisstring = sigma_string(i, net->sigma)) != NULL) {
+ printf("%s ",thisstring);
+ } else {
+ break;
+ }
+ }
+
+ printf("\n");
+
+ for (i = 0; i < maxsigma ; i++) {
+ for (j = 0; j < maxsigma ; j++) {
+ if (j == 0) {
+ if (i == 0) {
+ printf("%*s",lsymbol+1,"0");
+ printf("%*s",2,"*");
+ } else {
+ printf("%*s",lsymbol+1, sigma_string(i, net->sigma));
+ printf("%*d",2,*(cm+i*maxsigma+j));
+ }
+ j++;
+ j++;
+ continue;
+ }
+ if (i == j) {
+ printf("%.*s",(int)strlen(sigma_string(j, net->sigma))+1,"*");
+ } else {
+ printf("%.*d",(int)strlen(sigma_string(j, net->sigma))+1,*(cm+i*maxsigma+j));
+ }
+ }
+ printf("\n");
+ if (i == 0) {
+ i++; i++;
+ }
+ }
+}
+
+void cmatrix_init(struct fsm *net) {
+ int maxsigma, *cm, i, j;
+ if (net->medlookup == NULL) {
+ net->medlookup = xxcalloc(1,sizeof(struct medlookup));
+ }
+ maxsigma = sigma_max(net->sigma)+1;
+ cm = xxcalloc(maxsigma*maxsigma, sizeof(int));
+ net->medlookup->confusion_matrix = cm;
+ for (i = 0; i < maxsigma; i++) {
+ for (j = 0; j < maxsigma; j++) {
+ if (i == j)
+ *(cm+i*maxsigma+j) = 0;
+ else
+ *(cm+i*maxsigma+j) = 1;
+ }
+ }
+}
+
+void cmatrix_default_substitute(struct fsm *net, int cost) {
+ int i, j, maxsigma, *cm;
+ cm = net->medlookup->confusion_matrix;
+ maxsigma = sigma_max(net->sigma)+1;
+ for (i = 1; i < maxsigma; i++) {
+ for (j = 1; j < maxsigma; j++) {
+ if (i == j) {
+ *(cm+i*maxsigma+j) = 0;
+ } else {
+ *(cm+i*maxsigma+j) = cost;
+ }
+ }
+ }
+}
+
+void cmatrix_default_insert(struct fsm *net, int cost) {
+ int i, maxsigma, *cm;
+ cm = net->medlookup->confusion_matrix;
+ maxsigma = sigma_max(net->sigma)+1;
+ for (i = 0; i < maxsigma; i++) {
+ *(cm+i) = cost;
+ }
+}
+
+void cmatrix_default_delete(struct fsm *net, int cost) {
+ int i, maxsigma, *cm;
+ cm = net->medlookup->confusion_matrix;
+ maxsigma = sigma_max(net->sigma)+1;
+ for (i = 0; i < maxsigma; i++) {
+ *(cm+i*maxsigma) = cost;
+ }
+}
+
+void cmatrix_set_cost(struct fsm *net, char *in, char *out, int cost) {
+ int i, o, maxsigma, *cm;
+ cm = net->medlookup->confusion_matrix;
+ maxsigma = sigma_max(net->sigma) + 1;
+ if (in == NULL) {
+ i = 0;
+ } else {
+ i = sigma_find(in, net->sigma);
+ }
+ if (out == NULL) {
+ o = 0;
+ } else {
+ o = sigma_find(out, net->sigma);
+ }
+ if (i == -1) {
+ printf("Warning, symbol '%s' not in alphabet\n",in);
+ return;
+ }
+ if (o == -1) {
+ printf("Warning, symbol '%s' not in alphabet\n",out);
+ return;
+ }
+ *(cm+i*maxsigma+o) = cost;
+}
diff --git a/stack.c b/stack.c
new file mode 100644
index 0000000..9fd8f8c
--- /dev/null
+++ b/stack.c
@@ -0,0 +1,221 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include "foma.h"
+
+extern int quiet_mode;
+
+struct stack_entry *main_stack;
+
+int stack_size() {
+ int i;
+ struct stack_entry *stack_ptr;
+ for (i=0, stack_ptr = main_stack; stack_ptr->next != NULL; i++)
+ stack_ptr = stack_ptr->next;
+ return i;
+}
+
+int stack_init() {
+ main_stack = xxmalloc(sizeof(struct stack_entry));
+ main_stack->number = -1;
+ main_stack->fsm = NULL;
+ main_stack->next = NULL;
+ main_stack->previous = NULL;
+ return 1;
+}
+
+int stack_add(struct fsm *fsm) {
+ int i;
+ struct stack_entry *stack_ptr, *stack_ptr_previous;
+ stack_ptr_previous = NULL;
+
+ fsm_count(fsm);
+ if (strcmp(fsm->name,"") == 0)
+ sprintf(fsm->name, "%X",rand());
+ for (i=0, stack_ptr = main_stack; stack_ptr->number != -1; i++) {
+ stack_ptr_previous = stack_ptr;
+ stack_ptr = stack_ptr->next;
+ }
+ stack_ptr->next = xxmalloc(sizeof(struct stack_entry));
+ stack_ptr->fsm = fsm;
+ stack_ptr->ah = NULL;
+ stack_ptr->amedh = NULL;
+ stack_ptr->number = i;
+ stack_ptr->previous = stack_ptr_previous;
+ (stack_ptr->next)->number = -1;
+ (stack_ptr->next)->fsm = NULL;
+ (stack_ptr->next)->next = NULL;
+ (stack_ptr->next)->previous = stack_ptr;
+ if (!quiet_mode)
+ print_stats(fsm);
+ return(stack_ptr->number);
+}
+
+struct apply_med_handle *stack_get_med_ah() {
+ struct stack_entry *se;
+ se = stack_find_top();
+ if (se == NULL) {
+ return(NULL);
+ }
+ if (se->amedh == NULL) {
+ se->amedh = apply_med_init(se->fsm);
+ apply_med_set_align_symbol(se->amedh, "-");
+ }
+ return(se->amedh);
+}
+
+struct apply_handle *stack_get_ah() {
+ struct stack_entry *se;
+ se = stack_find_top();
+ if (se == NULL) {
+ return(NULL);
+ }
+ if (se->ah == NULL) {
+ se->ah = apply_init(se->fsm);
+ }
+ return(se->ah);
+}
+
+struct fsm *stack_pop(void) {
+ int i;
+ struct fsm *fsm;
+ struct stack_entry *stack_ptr;
+ if (stack_size() == 1) {
+ fsm = main_stack->fsm;
+ main_stack->fsm = NULL;
+ stack_clear();
+ return(fsm);
+ }
+ for (i=0, stack_ptr = main_stack; (stack_ptr->next)->number != -1; stack_ptr = stack_ptr->next, i++);
+ (stack_ptr->previous)->next = stack_ptr->next;
+ (stack_ptr->next)->previous = stack_ptr->previous;
+ fsm = stack_ptr->fsm;
+ if (stack_ptr->ah != NULL) {
+ apply_clear(stack_ptr->ah);
+ stack_ptr->ah = NULL;
+ }
+ if (stack_ptr->amedh != NULL) {
+ apply_med_clear(stack_ptr->amedh);
+ stack_ptr->amedh = NULL;
+ }
+ stack_ptr->fsm = NULL;
+ xxfree(stack_ptr);
+ return(fsm);
+}
+
+int stack_isempty () {
+ if (main_stack->next == NULL) {
+ return 1;
+ } else {
+ return 0;
+ }
+}
+
+int stack_turn () {
+ struct stack_entry *stack_ptr;
+ if (stack_isempty()) {
+ printf("Stack is empty.\n");
+ return 0;
+ }
+ if (stack_size() == 1) {
+ return 1;
+ }
+
+ stack_ptr = stack_find_top();
+ main_stack->next = stack_ptr->next;
+ (stack_ptr->next)->previous = main_stack;
+ main_stack = stack_ptr;
+
+ while (stack_ptr->previous != NULL) {
+ stack_ptr->next = stack_ptr->previous;
+ stack_ptr = stack_ptr->next;
+ }
+ for (stack_ptr = main_stack; stack_ptr->number != -1;) {
+ (stack_ptr->next)->previous = stack_ptr;
+ }
+ return 1;
+}
+
+struct stack_entry *stack_find_top () {
+ struct stack_entry *stack_ptr;
+ if (main_stack->number == -1) {
+ return NULL;
+ }
+ for (stack_ptr = main_stack; (stack_ptr->next)->number != -1; stack_ptr = stack_ptr->next);
+ return(stack_ptr);
+}
+
+struct stack_entry *stack_find_bottom () {
+ if (main_stack->number == -1) {
+ return NULL;
+ }
+ return(main_stack);
+}
+
+struct stack_entry *stack_find_second () {
+ struct stack_entry *stack_ptr;
+ /*
+ if (main_stack->number == -1) {
+ return NULL;
+ }
+ */
+ for (stack_ptr = main_stack; (stack_ptr->next)->number != -1; stack_ptr = stack_ptr->next);
+ return(stack_ptr->previous);
+}
+
+int stack_clear(void) {
+ struct stack_entry *stack_ptr;
+ for (stack_ptr = main_stack ; stack_ptr->next != NULL; stack_ptr = main_stack) {
+ if (stack_ptr->ah != NULL)
+ apply_clear(stack_ptr->ah);
+ if (stack_ptr->amedh != NULL)
+ apply_med_clear(stack_ptr->amedh);
+
+ main_stack = stack_ptr->next;
+ fsm_destroy(stack_ptr->fsm);
+ xxfree(stack_ptr);
+ }
+ xxfree(stack_ptr);
+ return(stack_init());
+}
+
+int stack_rotate () {
+
+ /* Top element of stack to bottom */
+ struct stack_entry *stack_ptr;
+ struct fsm *temp_fsm;
+
+ if (stack_isempty()) {
+ printf("Stack is empty.\n");
+ return -1;
+ }
+ if (stack_size() == 1) {
+ return 1;
+ }
+ stack_ptr = stack_find_top();
+ temp_fsm = main_stack->fsm;
+ main_stack->fsm = stack_ptr->fsm;
+ stack_ptr->fsm = temp_fsm;
+ return 1;
+}
+
+int stack_print () {
+ return 1;
+}
diff --git a/stringhash.c b/stringhash.c
new file mode 100644
index 0000000..93976ad
--- /dev/null
+++ b/stringhash.c
@@ -0,0 +1,101 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include "fomalib.h"
+
+#define STRING_HASH_SIZE 8191
+
+unsigned int sh_hashf(char *string);
+
+struct sh_handle *sh_init() {
+ struct sh_handle *sh;
+ sh = xxmalloc(sizeof(struct sh_handle));
+ sh->hash = xxcalloc(STRING_HASH_SIZE, sizeof(struct sh_hashtable));
+ return(sh);
+}
+
+void sh_done(struct sh_handle *sh) {
+ int i;
+ struct sh_hashtable *hash, *hashp;
+ for (i=0; i < STRING_HASH_SIZE; i++) {
+ hash = sh->hash + i;
+ if (hash->string != NULL)
+ xxfree(hash->string);
+ for (hash=hash->next ; hash != NULL ; hash = hashp) {
+ hashp = hash->next;
+ if (hash->string != NULL)
+ xxfree(hash->string);
+ xxfree(hash);
+ }
+ }
+ xxfree(sh->hash);
+ xxfree(sh);
+}
+
+int sh_get_value(struct sh_handle *sh) {
+ return(sh->lastvalue);
+}
+
+char *sh_find_string(struct sh_handle *sh, char *string) {
+ struct sh_hashtable *hash;
+ for (hash = sh->hash + sh_hashf(string) ; hash != NULL; hash = hash->next) {
+ if (hash->string == NULL)
+ return NULL;
+ if (strcmp(hash->string, string) == 0) {
+ sh->lastvalue = hash->value;
+ return(hash->string);
+ }
+ }
+ return NULL;
+}
+
+char *sh_find_add_string(struct sh_handle *sh, char *string, int value) {
+ char *s;
+ s = sh_find_string(sh, string);
+ if (s == NULL)
+ return (sh_add_string(sh, string, value));
+ else
+ return(s);
+}
+
+char *sh_add_string(struct sh_handle *sh, char *string, int value) {
+ struct sh_hashtable *hash, *newhash;
+
+ hash = sh->hash + sh_hashf(string);
+ if (hash->string == NULL) {
+ hash->string = xxstrdup(string);
+ hash->value = value;
+ return(hash->string);
+ } else {
+ newhash = xxmalloc(sizeof(struct sh_hashtable));
+ newhash->string = xxstrdup(string);
+ newhash->value = value;
+ newhash->next = hash->next;
+ hash->next = newhash;
+ return(newhash->string);
+ }
+}
+
+unsigned int sh_hashf(char *string) {
+ register unsigned int hash;
+ hash = 0;
+
+ while (*string != '\0') {
+ hash = hash * 101 + *string++;
+ }
+ return (hash % STRING_HASH_SIZE);
+}
diff --git a/structures.c b/structures.c
new file mode 100644
index 0000000..e44734f
--- /dev/null
+++ b/structures.c
@@ -0,0 +1,905 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <sys/time.h>
+#include "foma.h"
+
+static struct defined_quantifiers *quantifiers;
+
+char *fsm_get_library_version_string() {
+ static char s[20];
+ sprintf(s,"%i.%i.%i%s",MAJOR_VERSION,MINOR_VERSION,BUILD_VERSION,STATUS_VERSION);
+ return(s);
+}
+
+int linesortcompin(struct fsm_state *a, struct fsm_state *b) {
+ return (a->in - b->in);
+}
+
+int linesortcompout(struct fsm_state *a, struct fsm_state *b) {
+ return (a->out - b->out);
+}
+
+void fsm_sort_arcs(struct fsm *net, int direction) {
+ /* direction 1 = in, direction = 2, out */
+ struct fsm_state *fsm;
+ int i, lasthead, numlines;
+ int(*scin)() = linesortcompin;
+ int(*scout)() = linesortcompout;
+ fsm = net->states;
+ for (i=0, numlines = 0, lasthead = 0 ; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->state_no != (fsm+i+1)->state_no || (fsm+i)->target == -1) {
+ numlines++;
+ if ((fsm+i)->target == -1) {
+ numlines--;
+ }
+ if (numlines > 1) {
+ /* Sort, set numlines = 0 */
+ if (direction == 1)
+ qsort(fsm+lasthead, numlines, sizeof(struct fsm_state), scin);
+ else
+ qsort(fsm+lasthead, numlines, sizeof(struct fsm_state), scout);
+ }
+ numlines = 0;
+ lasthead = i + 1;
+ continue;
+ }
+ numlines++;
+ }
+ if (net->arity == 1) {
+ net->arcs_sorted_in = 1;
+ net->arcs_sorted_out = 1;
+ return;
+ }
+ if (direction == 1) {
+ net->arcs_sorted_in = 1;
+ net->arcs_sorted_out = 0;
+ }
+ if (direction == 2) {
+ net->arcs_sorted_out = 1;
+ net->arcs_sorted_in = 0;
+ }
+}
+
+struct state_array *map_firstlines(struct fsm *net) {
+ struct fsm_state *fsm;
+ struct state_array *sa;
+ int i, sold;
+ sold = -1;
+ sa = xxmalloc(sizeof(struct state_array)*((net->statecount)+1));
+ fsm = net->states;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->state_no != sold) {
+ (sa+((fsm+i)->state_no))->transitions = fsm+i;
+ sold = (fsm+i)->state_no;
+ }
+ }
+ return(sa);
+}
+
+struct fsm *fsm_boolean(int value) {
+ if (value == 0)
+ return (fsm_empty_set());
+ else
+ return(fsm_empty_string());
+}
+
+struct fsm *fsm_sigma_net(struct fsm *net) {
+ /* Extract sigma and create net with one arc */
+ /* from state 0 to state 1 with each (state 1 is final) */
+ struct sigma *sig;
+ int pathcount;
+
+ if (sigma_size(net->sigma) == 0) {
+ fsm_destroy(net);
+ return(fsm_empty_set());
+ }
+
+ fsm_state_init(sigma_max(net->sigma));
+ fsm_state_set_current_state(0, 0, 1);
+ pathcount = 0;
+ for (sig = net->sigma; sig != NULL; sig = sig->next) {
+ if (sig->number >=3 || sig->number == IDENTITY) {
+ pathcount++;
+ fsm_state_add_arc(0,sig->number, sig->number, 1, 0, 1);
+ }
+ }
+ fsm_state_end_state();
+ fsm_state_set_current_state(1, 1, 0);
+ fsm_state_end_state();
+ xxfree(net->states);
+ fsm_state_close(net);
+ net->is_minimized = YES;
+ net->is_loop_free = YES;
+ net->pathcount = pathcount;
+ sigma_cleanup(net, 1);
+ return(net);
+}
+
+struct fsm *fsm_sigma_pairs_net(struct fsm *net) {
+ /* Create FSM of attested pairs */
+ struct fsm_state *fsm;
+ char *pairs;
+ short int in, out;
+ int i, pathcount, smax;
+
+ smax = sigma_max(net->sigma)+1;
+ pairs = xxcalloc(smax*smax, sizeof(char));
+
+ fsm_state_init(sigma_max(net->sigma));
+ fsm_state_set_current_state(0, 0, 1);
+ pathcount = 0;
+ for (fsm = net->states, i=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target == -1)
+ continue;
+ in = (fsm+i)->in;
+ out = (fsm+i)->out;
+ if (*(pairs+smax*in+out) == 0) {
+ fsm_state_add_arc(0,in,out, 1, 0, 1);
+ *(pairs+smax*in+out) = 1;
+ pathcount++;
+ }
+ }
+ fsm_state_end_state();
+ fsm_state_set_current_state(1, 1, 0);
+ fsm_state_end_state();
+
+ xxfree(pairs);
+ xxfree(net->states);
+
+ fsm_state_close(net);
+ if (pathcount == 0) {
+ fsm_destroy(net);
+ return(fsm_empty_set());
+ }
+ net->is_minimized = YES;
+ net->is_loop_free = YES;
+ net->pathcount = pathcount;
+ sigma_cleanup(net, 1);
+ return(net);
+}
+
+int fsm_sigma_destroy(struct sigma *sigma) {
+ struct sigma *sig, *sigp;
+ for (sig = sigma, sigp = NULL; sig != NULL; sig = sigp) {
+ sigp = sig->next;
+ if (sig->symbol != NULL) {
+ xxfree(sig->symbol);
+ sig->symbol = NULL;
+ }
+ xxfree(sig);
+ }
+ return 1;
+}
+
+int fsm_destroy(struct fsm *net) {
+ if (net == NULL) {
+ return 0;
+ }
+ if (net->medlookup != NULL && net->medlookup->confusion_matrix != NULL) {
+ xxfree(net->medlookup->confusion_matrix);
+ net->medlookup->confusion_matrix = NULL;
+ }
+ if (net->medlookup != NULL) {
+ xxfree(net->medlookup);
+ net->medlookup = NULL;
+ }
+ fsm_sigma_destroy(net->sigma);
+ net->sigma = NULL;
+ if (net->states != NULL) {
+ xxfree(net->states);
+ net->states = NULL;
+ }
+ xxfree(net);
+ return(1);
+}
+
+struct fsm *fsm_create (char *name) {
+ struct fsm *fsm;
+ fsm = xxmalloc(sizeof(struct fsm));
+ strcpy(fsm->name, name);
+ fsm->arity = 1;
+ fsm->arccount = 0;
+ fsm->is_deterministic = NO;
+ fsm->is_pruned = NO;
+ fsm->is_minimized = NO;
+ fsm->is_epsilon_free = NO;
+ fsm->is_loop_free = NO;
+ fsm->arcs_sorted_in = NO;
+ fsm->arcs_sorted_out = NO;
+ fsm->sigma = sigma_create();
+ fsm->states = NULL;
+ fsm->medlookup = NULL;
+ return(fsm);
+}
+
+struct fsm *fsm_empty_string() {
+ struct fsm *net;
+ net = fsm_create("");
+ net->states = xxmalloc(sizeof(struct fsm_state)*2);
+ add_fsm_arc(net->states, 0, 0, -1, -1, -1, 1, 1);
+ add_fsm_arc(net->states, 1, -1, -1, -1, -1, -1, -1);
+ fsm_update_flags(net,YES,YES,YES,YES,YES,NO);
+ net->statecount = 1;
+ net->finalcount = 1;
+ net->arccount = 0;
+ net->linecount = 2;
+ net->pathcount = 1;
+ return(net);
+}
+
+struct fsm *fsm_identity() {
+ struct fsm *net;
+ struct sigma *sigma;
+ net = fsm_create("");
+ xxfree(net->sigma);
+ net->states = xxmalloc(sizeof(struct fsm_state)*3);
+ add_fsm_arc(net->states, 0, 0, 2, 2, 1, 0, 1);
+ add_fsm_arc(net->states, 1, 1, -1, -1, -1, 1, 0);
+ add_fsm_arc(net->states, 2, -1, -1, -1, -1, -1, -1);
+ sigma = xxmalloc(sizeof(struct sigma));
+ sigma->number = IDENTITY;
+ sigma->symbol = xxstrdup("@_IDENTITY_SYMBOL_@");
+ sigma->next = NULL;
+ net->sigma = sigma;
+ fsm_update_flags(net,YES,YES,YES,YES,YES,NO);
+ net->statecount = 2;
+ net->finalcount = 1;
+ net->arccount = 1;
+ net->linecount = 3;
+ net->pathcount = 1;
+ return(net);
+}
+
+struct fsm *fsm_empty_set() {
+ struct fsm *net;
+ net = fsm_create("");
+ net->states = fsm_empty();
+ fsm_update_flags(net,YES,YES,YES,YES,YES,NO);
+ net->statecount = 1;
+ net->finalcount = 0;
+ net->arccount = 0;
+ net->linecount = 2;
+ net->pathcount = 0;
+ return(net);
+}
+
+struct fsm_state *fsm_empty() {
+ struct fsm_state *new_fsm;
+ new_fsm = xxmalloc(sizeof(struct fsm_state)*2);
+ add_fsm_arc(new_fsm, 0, 0, -1, -1, -1, 0, 1);
+ add_fsm_arc(new_fsm, 1, -1, -1, -1, -1, -1, -1);
+ return(new_fsm);
+}
+
+int fsm_isuniversal(struct fsm *net) {
+ struct fsm_state *fsm;
+ net = fsm_minimize(net);
+ fsm_compact(net);
+ fsm = net->states;
+ if ((fsm->target == 0 && fsm->final_state == 1 && (fsm+1)->state_no == 0) &&
+ (fsm->in == IDENTITY && fsm->out == IDENTITY) &&
+ ((fsm+1)->state_no == -1) &&
+ (sigma_max(net->sigma)<3) ) {
+ return 1;
+ } else {
+ return 0;
+ }
+}
+
+int fsm_isempty(struct fsm *net) {
+ struct fsm_state *fsm;
+ net = fsm_minimize(net);
+ fsm = net->states;
+ if (fsm->target == -1 && fsm->final_state == 0 && (fsm+1)->state_no == -1)
+ return 1;
+ else
+ return 0;
+}
+
+int fsm_issequential(struct fsm *net) {
+ int i, *sigtable, sequential, seentrans, epstrans, laststate, insym;
+ struct fsm_state *fsm;
+ sigtable = xxcalloc(sigma_max(net->sigma)+1,sizeof(int));
+ for (i = 0 ; i < sigma_max(net->sigma)+1; i++) {
+ sigtable[i] = -2;
+ }
+ fsm = net->states;
+ seentrans = epstrans = 0;
+ laststate = -1;
+ for (sequential = 1, i = 0; (fsm+i)->state_no != -1 ; i++) {
+ insym = (fsm+i)->in;
+ if (insym < 0) {
+ continue;
+ }
+ if ((fsm+i)->state_no != laststate) {
+ laststate = (fsm+i)->state_no;
+ epstrans = 0;
+ seentrans = 0;
+ }
+ if (*(sigtable+insym) == laststate || epstrans == 1) {
+ sequential = 0;
+ break;
+ }
+ if (insym == EPSILON) {
+ if (epstrans == 1 || seentrans == 1) {
+ sequential = 0;
+ break;
+ }
+ epstrans = 1;
+ }
+ *(sigtable+insym) = laststate;
+ seentrans = 1;
+ }
+ xxfree(sigtable);
+ if (!sequential)
+ printf("fails at state %i\n",(fsm+i)->state_no);
+ return(sequential);
+}
+
+int fsm_isfunctional(struct fsm *net) {
+ return(fsm_isidentity(fsm_minimize(fsm_compose(fsm_invert(fsm_copy(net)),fsm_copy(net)))));
+}
+
+int fsm_isunambiguous(struct fsm *net) {
+ struct fsm *loweruniqnet, *testnet;
+ int ret;
+ loweruniqnet = fsm_lowerdet(fsm_copy(net));
+ testnet = fsm_minimize(fsm_compose(fsm_invert(fsm_copy(loweruniqnet)),fsm_copy(loweruniqnet)));
+ ret = fsm_isidentity(testnet);
+ fsm_destroy(loweruniqnet);
+ fsm_destroy(testnet);
+ return(ret);
+}
+
+struct fsm *fsm_extract_ambiguous_domain(struct fsm *net) {
+ // define AmbiguousDom(T) [_loweruniq(T) .o. _notid(_loweruniq(T).i .o. _loweruniq(T))].u;
+ struct fsm *loweruniqnet, *result;
+ loweruniqnet = fsm_lowerdet(net);
+ result = fsm_topsort(fsm_minimize(fsm_upper(fsm_compose(fsm_copy(loweruniqnet),fsm_extract_nonidentity(fsm_compose(fsm_invert(fsm_copy(loweruniqnet)), fsm_copy(loweruniqnet)))))));
+ fsm_destroy(loweruniqnet);
+ sigma_cleanup(result,1);
+ fsm_compact(result);
+ sigma_sort(result);
+ return(result);
+}
+
+struct fsm *fsm_extract_ambiguous(struct fsm *net) {
+ struct fsm *result;
+ result = fsm_topsort(fsm_minimize(fsm_compose(fsm_extract_ambiguous_domain(fsm_copy(net)), net)));
+ return(result);
+}
+
+struct fsm *fsm_extract_unambiguous(struct fsm *net) {
+ struct fsm *result;
+ result = fsm_topsort(fsm_minimize(fsm_compose(fsm_complement(fsm_extract_ambiguous_domain(fsm_copy(net))), net)));
+ return(result);
+}
+
+int fsm_isidentity(struct fsm *net) {
+
+ /* We check whether a given transducer only produces identity relations */
+ /* By doing a DFS on the graph, and storing, for each state a "discrepancy" */
+ /* string, showing the current "debt" on the upper or lower side. */
+ /* We immediately fail if: */
+ /* a) we encounter an already seen state with a different current */
+ /* discrepancy than what is stored in the state. */
+ /* b) when traversing an arc, we encounter a mismatch between the arc and */
+ /* the current discrepancy. */
+ /* c) we encounter a final state and have a non-null current discrepancy. */
+ /* d) we encounter @ with a non-null discrepancy anywhere. */
+ /* e) we encounter ? anywhere. */
+
+ struct discrepancy {
+ short int *string;
+ short int length;
+ _Bool visited;
+ };
+
+ struct state_array *state_array;
+ struct fsm_state *curr_ptr;
+ int i, j, v, vp, num_states, factor = 0, newlength = 1, startfrom;
+ short int in, out, *newstring;
+ struct discrepancy *discrepancy, *currd, *targetd;
+
+ fsm_minimize(net);
+ fsm_count(net);
+
+ num_states = net->statecount;
+ discrepancy = xxcalloc(num_states,sizeof(struct discrepancy));
+ state_array = map_firstlines(net);
+ ptr_stack_clear();
+ ptr_stack_push(state_array->transitions);
+
+ while(!ptr_stack_isempty()) {
+
+ curr_ptr = ptr_stack_pop();
+
+ nopop:
+ v = curr_ptr->state_no; /* source state number */
+ vp = curr_ptr->target; /* target state number */
+ currd = discrepancy+v;
+ if (v != -1)
+ currd->visited = 1;
+ if (v == -1 || vp == -1)
+ continue;
+ in = curr_ptr->in;
+ out = curr_ptr->out;
+
+ targetd = discrepancy+vp;
+ /* Check arc and conditions e) d) b) */
+ /* e) */
+ if (in == UNKNOWN || out == UNKNOWN)
+ goto fail;
+ /* d) */
+ if (in == IDENTITY && currd->length != 0)
+ goto fail;
+ /* b) */
+ if (currd->length != 0) {
+ if (currd->length > 0 && out != EPSILON && out != *(currd->string))
+ goto fail;
+ if (currd->length < 0 && in != EPSILON && in != *(currd->string))
+ goto fail;
+ }
+ if (currd->length == 0 && in != out && in != EPSILON && out != EPSILON) {
+ goto fail;
+ }
+
+ /* Calculate new discrepancy */
+ if (currd->length != 0) {
+ if (in != EPSILON && out != EPSILON)
+ factor = 0;
+ else if (in == EPSILON)
+ factor = -1;
+ else if (out == EPSILON)
+ factor = 1;
+
+ newlength = currd->length + factor;
+ startfrom = (abs(newlength) <= abs(currd->length)) ? 1 : 0;
+
+ } else if (currd->length == 0) {
+ if (in != EPSILON && out != EPSILON) {
+ newlength = 0;
+ } else {
+ newlength = (out == EPSILON) ? 1 : -1;
+ }
+ startfrom = 0;
+ }
+
+ newstring = xxcalloc(abs(newlength),sizeof(int));
+
+ for (i = startfrom, j = 0; i < abs(currd->length); i++, j++) {
+ *(newstring+j) = *((currd->string)+i);
+ }
+ if (newlength != 0) {
+ if (currd->length > 0 && newlength >= currd->length) {
+ *(newstring+j) = in;
+ }
+ if (currd->length < 0 && newlength <= currd->length) {
+ *(newstring+j) = out;
+ }
+ if (currd->length == 0 && newlength < currd->length) {
+ *(newstring+j) = out;
+ }
+ if (currd->length == 0 && newlength > currd->length) {
+ *(newstring+j) = in;
+ }
+ }
+
+ /* Check target conditions a) c) */
+ /* a) */
+ if (((state_array+vp)->transitions)->final_state && newlength != 0)
+ goto fail;
+ if (curr_ptr->state_no == (curr_ptr+1)->state_no) {
+ ptr_stack_push(curr_ptr+1);
+ }
+ if ((discrepancy+vp)->visited) {
+ //xxfree(newstring);
+ if (targetd->length != newlength)
+ goto fail;
+ for (i=0 ; i < abs(newlength); i++) {
+ if (*((targetd->string)+i) != *(newstring+i))
+ goto fail;
+ }
+ } else {
+ /* Add discrepancy to target state */
+ targetd->length = newlength;
+ targetd->string = newstring;
+ curr_ptr = (state_array+vp)->transitions;
+ goto nopop;
+ }
+ }
+ xxfree(state_array);
+ xxfree(discrepancy);
+ return 1;
+ fail:
+ xxfree(state_array);
+ xxfree(discrepancy);
+ ptr_stack_clear();
+ return 0;
+}
+
+struct fsm *fsm_markallfinal(struct fsm *net) {
+ struct fsm_state *fsm;
+ int i;
+ fsm = net->states;
+ for (i=0; (fsm+i)->state_no != -1; i++) {
+ (fsm+i)->final_state = YES;
+ }
+ return net;
+}
+
+struct fsm *fsm_lowerdet(struct fsm *net) {
+ unsigned int newsym; /* Running number for new syms */
+ struct fsm_state *fsm;
+ char repstr[13];
+ int i,j,maxsigma,maxarc;
+ net = fsm_minimize(net);
+ fsm_count(net);
+ newsym = 8723643;
+ fsm = net->states;
+ maxarc = 0;
+ maxsigma = sigma_max(net->sigma);
+
+ for (i=0, j=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1)
+ j++;
+ if ((fsm+i+1)->state_no != (fsm+i)->state_no) {
+ maxarc = maxarc > j ? maxarc : j;
+ j = 0;
+ }
+ }
+ if (maxarc > (maxsigma-2)) {
+ for (i=maxarc; i > (maxsigma-2); i--) {
+ sprintf(repstr,"%012X",newsym++);
+ sigma_add(repstr, net->sigma);
+ }
+ sigma_sort(net);
+ }
+ for (i=0, j=3; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1) {
+ (fsm+i)->out = j++;
+ (fsm+i)->in = ((fsm+i)->in == IDENTITY) ? UNKNOWN : (fsm+i)->in;
+ }
+ if ((fsm+i+1)->state_no != (fsm+i)->state_no) {
+ j = 3;
+ }
+ }
+ return(net);
+}
+
+struct fsm *fsm_lowerdeteps(struct fsm *net) {
+ unsigned int newsym; /* Running number for new syms */
+ struct fsm_state *fsm;
+ char repstr[13];
+ int i,j,maxsigma,maxarc;
+ net = fsm_minimize(net);
+ fsm_count(net);
+ newsym = 8723643;
+ fsm = net->states;
+ maxarc = 0;
+ maxsigma = sigma_max(net->sigma);
+
+ for (i=0, j=0; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1)
+ j++;
+ if ((fsm+i+1)->state_no != (fsm+i)->state_no) {
+ maxarc = maxarc > j ? maxarc : j;
+ j = 0;
+ }
+ }
+ if (maxarc > (maxsigma-2)) {
+ for (i=maxarc; i > (maxsigma-2); i--) {
+ sprintf(repstr,"%012X",newsym++);
+ sigma_add(repstr, net->sigma);
+ }
+ sigma_sort(net);
+ }
+ for (i=0, j=3; (fsm+i)->state_no != -1; i++) {
+ if ((fsm+i)->target != -1 && (fsm+i)->out != EPSILON) {
+ (fsm+i)->out = j++;
+ (fsm+i)->in = ((fsm+i)->in == IDENTITY) ? UNKNOWN : (fsm+i)->in;
+ }
+ if ((fsm+i+1)->state_no != (fsm+i)->state_no) {
+ j = 3;
+ }
+ }
+ return(net);
+}
+
+struct fsm *fsm_extract_nonidentity(struct fsm *net) {
+
+ /* Same algorithm as for test identity, except we mark the arcs that cause nonidentity */
+ /* Experimental. */
+
+ struct discrepancy {
+ short int *string;
+ short int length;
+ _Bool visited;
+ };
+
+ struct state_array *state_array;
+ struct fsm_state *curr_ptr;
+ struct fsm *net2;
+ int i, j, v, vp, num_states, factor = 0, newlength = 1, startfrom, killnum;
+ short int in, out, *newstring;
+ struct discrepancy *discrepancy, *currd, *targetd;
+
+ fsm_minimize(net);
+ fsm_count(net);
+ killnum = sigma_add("@KILL@", net->sigma);
+
+ num_states = net->statecount;
+ discrepancy = xxcalloc(num_states,sizeof(struct discrepancy));
+ state_array = map_firstlines(net);
+ ptr_stack_push(state_array->transitions);
+
+ while(!ptr_stack_isempty()) {
+
+ curr_ptr = ptr_stack_pop();
+
+ nopop:
+ v = curr_ptr->state_no; /* source state number */
+ vp = curr_ptr->target; /* target state number */
+ currd = discrepancy+v;
+ if (v != -1)
+ currd->visited = 1;
+ if (v == -1 || vp == -1)
+ continue;
+ in = curr_ptr->in;
+ out = curr_ptr->out;
+
+ targetd = discrepancy+vp;
+ /* Check arc and conditions e) d) b) */
+ /* e) */
+ if (in == UNKNOWN || out == UNKNOWN)
+ goto fail;
+ /* d) */
+ if (in == IDENTITY && currd->length != 0)
+ goto fail;
+ /* b) */
+ if (currd->length != 0) {
+ if (currd->length > 0 && out != EPSILON && out != *(currd->string))
+ goto fail;
+ if (currd->length < 0 && in != EPSILON && in != *(currd->string))
+ goto fail;
+ }
+ if (currd->length == 0 && in != out && in != EPSILON && out != EPSILON) {
+ goto fail;
+ }
+
+ /* Calculate new discrepancy */
+ if (currd->length != 0) {
+ if (in != EPSILON && out != EPSILON)
+ factor = 0;
+ else if (in == EPSILON)
+ factor = -1;
+ else if (out == EPSILON)
+ factor = 1;
+
+ newlength = currd->length + factor;
+ startfrom = (abs(newlength) <= abs(currd->length)) ? 1 : 0;
+
+ } else if (currd->length == 0) {
+ if (in != EPSILON && out != EPSILON) {
+ newlength = 0;
+ } else {
+ newlength = (out == EPSILON) ? 1 : -1;
+ }
+ startfrom = 0;
+ }
+
+ newstring = xxcalloc(abs(newlength),sizeof(int));
+
+ for (i = startfrom, j = 0; i < abs(currd->length); i++, j++) {
+ *(newstring+j) = *((currd->string)+i);
+ }
+ if (newlength != 0) {
+ if (currd->length > 0 && newlength >= currd->length) {
+ *(newstring+j) = in;
+ }
+ if (currd->length < 0 && newlength <= currd->length) {
+ *(newstring+j) = out;
+ }
+ if (currd->length == 0 && newlength < currd->length) {
+ *(newstring+j) = out;
+ }
+ if (currd->length == 0 && newlength > currd->length) {
+ *(newstring+j) = in;
+ }
+
+ }
+
+ /* Check target conditions a) c) */
+ /* a) */
+ if (((state_array+vp)->transitions)->final_state && newlength != 0)
+ goto fail;
+ if (curr_ptr->state_no == (curr_ptr+1)->state_no) {
+ ptr_stack_push(curr_ptr+1);
+ }
+
+ if ((discrepancy+vp)->visited) {
+ //xxfree(newstring);
+ if (targetd->length != newlength)
+ goto fail;
+ for (i=0 ; i < abs(newlength); i++) {
+ if (*((targetd->string)+i) != *(newstring+i))
+ goto fail;
+ }
+ } else {
+ /* Add discrepancy to target state */
+ targetd->length = newlength;
+ targetd->string = newstring;
+ curr_ptr = (state_array+vp)->transitions;
+ goto nopop;
+ }
+ continue;
+ fail:
+ curr_ptr->out = killnum;
+ if (curr_ptr->state_no == (curr_ptr+1)->state_no) {
+ ptr_stack_push(curr_ptr+1);
+ }
+ }
+ ptr_stack_clear();
+ sigma_sort(net);
+ net2 = fsm_upper(fsm_compose(net,fsm_contains(fsm_symbol("@KILL@"))));
+ sigma_remove("@KILL@",net2->sigma);
+ sigma_sort(net2);
+ xxfree(state_array);
+ xxfree(discrepancy);
+ return(net2);
+}
+
+struct fsm *fsm_copy (struct fsm *net) {
+ struct fsm *net_copy;
+ if (net == NULL)
+ return net;
+
+ net_copy = xxmalloc(sizeof(struct fsm));
+ memcpy(net_copy, net, sizeof(struct fsm));
+
+ fsm_count(net);
+ net_copy->sigma = sigma_copy(net->sigma);
+ net_copy->states = fsm_state_copy(net->states, net->linecount);
+ return(net_copy);
+}
+
+struct fsm_state *fsm_state_copy(struct fsm_state *fsm_state, int linecount) {
+ struct fsm_state *new_fsm_state;
+
+ new_fsm_state = xxmalloc(sizeof(struct fsm_state)*(linecount));
+ memcpy(new_fsm_state, fsm_state, linecount*sizeof(struct fsm_state));
+ return(new_fsm_state);
+}
+
+/* TODO: separate linecount and arccount */
+int find_arccount(struct fsm_state *fsm) {
+ int i;
+ for (i=0;(fsm+i)->state_no != -1; i++) {
+ }
+ return i;
+}
+
+void clear_quantifiers() {
+ quantifiers = NULL;
+}
+
+int count_quantifiers() {
+ struct defined_quantifiers *q;
+ int i;
+ for (i = 0, q = quantifiers; q != NULL; q = q->next) {
+ i++;
+ }
+ return(i);
+}
+
+void add_quantifier (char *string) {
+ struct defined_quantifiers *q;
+ if (quantifiers == NULL) {
+ q = xxmalloc(sizeof(struct defined_quantifiers));
+ quantifiers = q;
+ } else {
+ for (q = quantifiers; q->next != NULL; q = q->next) {
+
+ }
+ q->next = xxmalloc(sizeof(struct defined_quantifiers));
+ q = q->next;
+ }
+ q->name = xxstrdup(string);
+ q->next = NULL;
+}
+
+struct fsm *union_quantifiers() {
+/* We create a FSM that simply accepts the union of all */
+/* quantifier symbols */
+
+ struct fsm *net;
+ struct defined_quantifiers *q;
+ int i, syms, s, symlo;
+
+ net = fsm_create("");
+ fsm_update_flags(net, YES, YES, YES, YES, NO, NO);
+
+ for (syms = 0, symlo = 0, q = quantifiers; q != NULL; q = q->next) {
+ s = sigma_add(q->name, net->sigma);
+ if (symlo == 0) {
+ symlo = s;
+ }
+ syms++;
+ }
+ net->states = xxmalloc(sizeof(struct fsm_state)*(syms+1));
+ for (i = 0; i < syms; i++) {
+ add_fsm_arc(net->states, i, 0, symlo+i, symlo+i, 0, 1, 1);
+ }
+ add_fsm_arc(net->states, i, -1, -1, -1, -1 ,-1 ,-1);
+ net->arccount = syms;
+ net->statecount = net->finalcount = 1;
+ net->linecount = syms;
+ return(net);
+}
+
+char *find_quantifier (char *string) {
+ struct defined_quantifiers *q;
+ for (q = quantifiers; q != NULL; q = q->next) {
+ if (strcmp(string,q->name) == 0)
+ return q->name;
+ }
+ return(NULL);
+}
+
+void purge_quantifier (char *string) {
+ struct defined_quantifiers *q, *q_prev;
+ for (q = quantifiers, q_prev = NULL; q != NULL; q_prev = q, q = q->next) {
+ if (strcmp(string, q->name) == 0) {
+ if (q_prev != NULL) {
+ q_prev->next = q->next;
+ } else {
+ quantifiers = q->next;
+ }
+ }
+ }
+}
+
+struct fsm *fsm_quantifier(char *string) {
+
+ /* \x* x \x* x \x* */
+ return(fsm_concat(fsm_kleene_star(fsm_term_negation(fsm_symbol(string))),fsm_concat(fsm_symbol(string),fsm_concat(fsm_kleene_star(fsm_term_negation(fsm_symbol(string))),fsm_concat(fsm_symbol(string),fsm_kleene_star(fsm_term_negation(fsm_symbol(string))))))));
+
+}
+
+struct fsm *fsm_logical_precedence(char *string1, char *string2) {
+ /* x < y = \y* x \y* [x | y Q* x] ?* */
+ /* 1 2 3 4 5 */
+
+ return(fsm_concat(fsm_kleene_star(fsm_term_negation(fsm_symbol(string2))),fsm_concat(fsm_symbol(string1),fsm_concat(fsm_kleene_star(fsm_term_negation(fsm_symbol(string2))),fsm_concat(fsm_union(fsm_symbol(string1),fsm_concat(fsm_symbol(string2),fsm_concat(union_quantifiers(),fsm_symbol(string1)))),fsm_universal())))));
+
+/* 1,3 fsm_kleene_star(fsm_term_negation(fsm_symbol(string2))) */
+/* 2 = fsm_symbol(string1) */
+/* 5 = fsm_universal() */
+/* 4 = fsm_union(fsm_symbol(string1),fsm_concat(fsm_symbol(string2),fsm_concat(union_quantifiers(),fsm_symbol(string1)))) */
+
+}
+
+/** Logical equivalence, i.e. where two variables span the same substring */
+/** x = y = ?* [x y|y x]/Q ?* [x y|y x]/Q ?* */
+struct fsm *fsm_logical_eq(char *string1, char *string2) {
+ return(fsm_concat(fsm_universal(),fsm_concat(fsm_ignore(fsm_union(fsm_concat(fsm_symbol(string1),fsm_symbol(string2)),fsm_concat(fsm_symbol(string2),fsm_symbol(string1))),union_quantifiers(),OP_IGNORE_ALL),fsm_concat(fsm_universal(),fsm_concat(fsm_ignore(fsm_union(fsm_concat(fsm_symbol(string1),fsm_symbol(string2)),fsm_concat(fsm_symbol(string2),fsm_symbol(string1))),union_quantifiers(),OP_IGNORE_ALL),fsm_universal())))));
+}
+
diff --git a/topsort.c b/topsort.c
new file mode 100644
index 0000000..086620a
--- /dev/null
+++ b/topsort.c
@@ -0,0 +1,170 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <assert.h>
+#include "foma.h"
+
+struct fsm *fsm_topsort (struct fsm *net) {
+
+ /* We topologically sort the network by looking for a state */
+ /* with inverse count 0. We then examine all the arcs from that */
+ /* state, and decrease the target invcounts. If we find a new */
+ /* state with invcount 0, we push that on the stack to be treated */
+ /* If the graph is cyclic, one of two things will happen: */
+
+ /* (1) We fail to find a state with invcount 0 before we've treated */
+ /* all states */
+ /* (2) A state under treatment has an arc to a state already treated */
+ /* or itself (we mark a state as treated as soon as we start */
+ /* working on it). */
+ /* Of course we also count the number of paths in the network. */
+
+ int i, j, curr_state, *statemap, treatcount, *order, lc, *newnum, newtarget, newstate;
+ unsigned short int *invcount;
+ unsigned char *treated, overflow;
+ long long grand_pathcount, *pathcount;
+ struct fsm_state *fsm, *curr_fsm, *new_fsm;
+
+ fsm_count(net);
+
+ fsm = net->states;
+
+ statemap = xxmalloc(sizeof(int)*net->statecount);
+ order = xxmalloc(sizeof(int)*net->statecount);
+ pathcount = xxmalloc(sizeof(long long)*net->statecount);
+ newnum = xxmalloc(sizeof(int)*net->statecount);
+ invcount = xxmalloc(sizeof(unsigned short int)*net->statecount);
+ treated = xxmalloc(sizeof(unsigned char)*net->statecount);
+
+ for (i=0; i < net->statecount; i++) {
+ *(statemap+i) = -1;
+ *(invcount+i) = 0;
+ *(treated+i) = 0;
+ *(order+i) = 0;
+ *(pathcount+i) = 0;
+ }
+
+ for (i=0, lc=0; (fsm+i)->state_no != -1; i++) {
+ lc++;
+ if ((fsm+i)->target != -1) {
+ (*(invcount+(fsm+i)->target))++;
+ /* Do a fast check here to see if we have a selfloop */
+ if ((fsm+i)->state_no == (fsm+i)->target) {
+ net->pathcount = PATHCOUNT_CYCLIC;
+ net->is_loop_free = 0;
+ goto cyclic;
+ }
+ }
+ if (*(statemap+(fsm+i)->state_no) == -1) {
+ *(statemap+(fsm+i)->state_no) = i;
+ }
+ }
+
+ treatcount = net->statecount;
+ int_stack_clear();
+ int_stack_push(0);
+ grand_pathcount = 0;
+
+ *(pathcount+0) = 1;
+
+ overflow = 0;
+ for (i=0 ; !int_stack_isempty(); i++) {
+ /* Treat a state */
+ curr_state = int_stack_pop();
+ *(treated+curr_state) = 1;
+ *(order+i) = curr_state;
+ *(newnum+curr_state) = i;
+
+ treatcount--;
+ curr_fsm = fsm+*(statemap+curr_state);
+ while (curr_fsm->state_no == curr_state) {
+ if (curr_fsm->target != -1 ) {
+ (*(invcount+(curr_fsm->target)))--;
+
+ /* Check if we overflow the path counter */
+
+ if (!overflow) {
+ *(pathcount+(curr_fsm->target)) += *(pathcount+curr_state);
+ if ((*(pathcount+(curr_fsm->target)) < 0)) {
+ overflow = 1;
+ }
+ }
+
+ /* Case (1) for cyclic */
+ if (*(treated+(curr_fsm)->target) == 1) {
+ net->pathcount = PATHCOUNT_CYCLIC;
+ net->is_loop_free = 0;
+ goto cyclic;
+ }
+ if ( *(invcount+(curr_fsm->target)) == 0) {
+ int_stack_push(curr_fsm->target);
+ }
+ }
+ curr_fsm++;
+ }
+ }
+
+ /* Case (2) */
+ if (treatcount > 0) {
+ net->pathcount = PATHCOUNT_CYCLIC;
+ net->is_loop_free = 0;
+ goto cyclic;
+ }
+
+ new_fsm = xxmalloc(sizeof(struct fsm_state) * (lc+1));
+ for (i=0, j=0 ; i < net->statecount; i++) {
+
+ curr_state = *(order+i);
+ curr_fsm = fsm+*(statemap+curr_state);
+
+ if (curr_fsm->final_state == 1 && !overflow) {
+ grand_pathcount += *(pathcount + curr_state);
+ if (grand_pathcount < 0)
+ overflow = 1;
+ }
+
+ for (; curr_fsm->state_no == curr_state; curr_fsm++) {
+
+ newstate = curr_fsm->state_no == -1 ? -1 : *(newnum+(curr_fsm->state_no));
+ newtarget = curr_fsm->target == -1 ? -1 : *(newnum+(curr_fsm->target));
+ add_fsm_arc(new_fsm, j, newstate, curr_fsm->in, curr_fsm->out, newtarget, curr_fsm->final_state, curr_fsm->start_state);
+ j++;
+ }
+ }
+
+ add_fsm_arc(new_fsm, j, -1, -1, -1, -1, -1, -1);
+ net->states = new_fsm;
+ net->pathcount = grand_pathcount;
+ net->is_loop_free = 1;
+ if (overflow == 1) {
+ net->pathcount = PATHCOUNT_OVERFLOW;
+ }
+ xxfree(fsm);
+
+ cyclic:
+
+ xxfree(statemap);
+ xxfree(order);
+ xxfree(pathcount);
+ xxfree(newnum);
+ xxfree(invcount);
+ xxfree(treated);
+ int_stack_clear();
+ return(net);
+}
diff --git a/trie.c b/trie.c
new file mode 100644
index 0000000..742a33f
--- /dev/null
+++ b/trie.c
@@ -0,0 +1,151 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include "fomalib.h"
+
+#define THASH_TABLESIZE 1048573
+#define TRIE_STATESIZE 32768
+
+unsigned int trie_hashf(unsigned int source, char *insym, char *outsym);
+
+struct fsm_trie_handle *fsm_trie_init() {
+ struct fsm_trie_handle *th;
+
+ th = xxcalloc(1,sizeof(struct fsm_trie_handle));
+ th->trie_hash = xxcalloc(THASH_TABLESIZE, sizeof(struct trie_hash));
+ th->trie_states = xxcalloc(TRIE_STATESIZE, sizeof(struct trie_states));
+ th->statesize = TRIE_STATESIZE;
+ th->trie_cursor = 0;
+ th->sh_hash = sh_init();
+ return(th);
+}
+
+struct fsm *fsm_trie_done(struct fsm_trie_handle *th) {
+ struct trie_hash *thash, *thashp;
+ struct fsm *newnet;
+ struct fsm_construct_handle *newh;
+ unsigned int i;
+
+ newh = fsm_construct_init("name");
+ for (i = 0; i < THASH_TABLESIZE; i++) {
+ thash = (th->trie_hash)+i;
+ for ( ; thash != NULL; thash = thash->next) {
+ if (thash->insym != NULL) {
+ fsm_construct_add_arc(newh, thash->sourcestate, thash->targetstate, thash->insym, thash->outsym);
+ } else {
+ break;
+ }
+ }
+ }
+ for (i = 0; i <= th->used_states; i++) {
+ if ((th->trie_states+i)->is_final == 1) {
+ fsm_construct_set_final(newh, i);
+ }
+ }
+ fsm_construct_set_initial(newh, 0);
+ newnet = fsm_construct_done(newh);
+ /* Free all mem */
+ for (i=0; i < THASH_TABLESIZE; i++) {
+ for (thash=((th->trie_hash)+i)->next; thash != NULL; thash = thashp) {
+ thashp = thash->next;
+ xxfree(thash);
+ }
+ }
+ sh_done(th->sh_hash);
+ xxfree(th->trie_states);
+ xxfree(th->trie_hash);
+ xxfree(th);
+ return(newnet);
+}
+
+void fsm_trie_add_word(struct fsm_trie_handle *th, char *word) {
+ int i, len;
+ char *wcopy;
+ wcopy = xxstrdup(word);
+ len = strlen(wcopy);
+ for (i=0 ; *word != '\0' && i < len; word = word + utf8skip(word)+1, i++) {
+ strncpy(wcopy, word, utf8skip(word)+1);
+ *(wcopy+utf8skip(word)+1) = '\0';
+ fsm_trie_symbol(th, wcopy, wcopy);
+ }
+ xxfree(wcopy);
+ fsm_trie_end_word(th);
+}
+
+void fsm_trie_end_word(struct fsm_trie_handle *th) {
+ (th->trie_states+th->trie_cursor)->is_final = 1;
+ th->trie_cursor = 0;
+}
+
+void fsm_trie_symbol(struct fsm_trie_handle *th, char *insym, char *outsym) {
+ unsigned int h;
+ struct trie_hash *thash, *newthash;
+
+ h = trie_hashf(th->trie_cursor, insym, outsym);
+ if ((th->trie_hash+h)->insym != NULL) {
+ for (thash = th->trie_hash+h; thash != NULL; thash = thash->next) {
+ if (strcmp(thash->insym, insym) == 0 && strcmp(thash->outsym, outsym) == 0 && thash->sourcestate == th->trie_cursor) {
+ /* Exists, move cursor */
+ th->trie_cursor = thash->targetstate;
+ return;
+ }
+ }
+ }
+ /* Doesn't exist */
+
+ /* Insert trans, move counter and cursor */
+ th->used_states++;
+ thash = th->trie_hash+h;
+ if (thash->insym == NULL) {
+ thash->insym = sh_find_add_string(th->sh_hash, insym,1);
+ thash->outsym = sh_find_add_string(th->sh_hash, outsym,1);
+ thash->sourcestate = th->trie_cursor;
+ thash->targetstate = th->used_states;
+ } else {
+ newthash = xxcalloc(1, sizeof(struct trie_hash));
+ newthash->next = thash->next;
+ newthash->insym = sh_find_add_string(th->sh_hash, insym,1);
+ newthash->outsym = sh_find_add_string(th->sh_hash, outsym,1);
+ newthash->sourcestate = th->trie_cursor;
+ newthash->targetstate = th->used_states;
+ thash->next = newthash;
+ }
+ th->trie_cursor = th->used_states;
+
+ /* Realloc */
+ if (th->used_states >= th->statesize) {
+ th->statesize = next_power_of_two(th->statesize);
+ th->trie_states = xxrealloc(th->trie_states, th->statesize * sizeof(struct trie_states));
+ }
+ (th->trie_states+th->used_states)->is_final = 0;
+}
+
+unsigned int trie_hashf(unsigned int source, char *insym, char *outsym) {
+
+ /* Hash based on insym, outsym, and sourcestate */
+ register unsigned int hash;
+ hash = 0;
+
+ while (*insym != '\0') {
+ hash = hash * 101 + *insym++;
+ }
+ while (*outsym != '\0') {
+ hash = hash * 101 + *outsym++;
+ }
+ hash = hash * 101 + source;
+ return (hash % THASH_TABLESIZE);
+}
diff --git a/utf8.c b/utf8.c
new file mode 100644
index 0000000..f0000e3
--- /dev/null
+++ b/utf8.c
@@ -0,0 +1,272 @@
+/* Foma: a finite-state toolkit and library. */
+/* Copyright © 2008-2015 Mans Hulden */
+
+/* This file is part of foma. */
+
+/* Licensed under the Apache License, Version 2.0 (the "License"); */
+/* you may not use this file except in compliance with the License. */
+/* You may obtain a copy of the License at */
+
+/* http://www.apache.org/licenses/LICENSE-2.0 */
+
+/* Unless required by applicable law or agreed to in writing, software */
+/* distributed under the License is distributed on an "AS IS" BASIS, */
+/* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. */
+/* See the License for the specific language governing permissions and */
+/* limitations under the License. */
+
+#include <stdlib.h>
+#include <string.h>
+#include "foma.h"
+
+unsigned char *int2utf8str(int codepoint);
+
+static int hexstrtoint(char *str);
+
+/* Removes trailing character c, as well as spaces and tabs */
+char *remove_trailing(char *s, char c) {
+ int i, len;
+ len = strlen(s)-1;
+ for (i = len; i>=0 ; i--) {
+ if (*(s+i) != c && *(s+i) != ' ' && *(s+i) != '\t') {
+ break;
+ }
+ *(s+i) = '\0';
+ }
+ return(s);
+}
+
+/* Remove trailing space and \t */
+char *trim(char *string) {
+ int i;
+ if (string == NULL)
+ return(string);
+ for (i = strlen(string) - 1; i >=0; i--) {
+ if (*(string+i) != ' ' && *(string+i) != '\t')
+ break;
+ *(string+i) = '\0';
+ }
+ return(string);
+}
+
+/* Reverses string in-place */
+char *xstrrev(char *str) {
+ char *p1, *p2;
+ if (! str || ! *str)
+ return str;
+ for (p1 = str, p2 = str + strlen(str) - 1; p2 > p1; ++p1, --p2) {
+ *p1 ^= *p2;
+ *p2 ^= *p1;
+ *p1 ^= *p2;
+ }
+ return str;
+}
+
+char *escape_string(char *string, char chr) {
+ size_t i,j;
+ char *newstring;
+ for (i=0,j=0; i < strlen(string); i++) {
+ if (string[i] == chr) {
+ j++;
+ }
+ }
+ if (j>0) {
+ newstring = xxcalloc((strlen(string)+j),sizeof(char));
+ for (i=0,j=0; i<strlen(string); i++, j++) {
+ if (string[i] == chr) {
+ newstring[j++] = '\\';
+ newstring[j] = chr;
+ } else {
+ newstring[j] = string[i];
+ }
+ }
+ return(newstring);
+ } else {
+ return(string);
+ }
+}
+
+/* Substitute first \n for \0 */
+void strip_newline(char *s) {
+ int i, len;
+ len = strlen(s);
+ /* remove the null terminator */
+ for (i = 0; i < len; i++ ) {
+ if (s[i] == '\n' ) {
+ s[i] = '\0';
+ return;
+ }
+ }
+}
+/* Removes initial and final quote, and decodes the string if it contains special chars */
+void dequote_string(char *s) {
+ int len, i, j;
+ len = strlen(s);
+ if (*s == 0x22 && *(s+len-1) == 0x22) {
+ for (i = 1, j = 0; i<len-1; i++, j++) {
+ *(s+j) = *(s+i);
+
+ }
+ *(s+j) = '\0';
+ decode_quoted(s);
+ }
+}
+
+/* Decode quoted strings. This includes: */
+/* Changing \uXXXX sequences to their unicode equivalents */
+
+void decode_quoted(char *s) {
+ int len, i, j, skip;
+ unsigned char *unistr;
+
+ len = strlen(s);
+ for (i=0, j=0; i < len; ) {
+ if (*(s+i) == 0x5c && len-i > 5 && *(s+i+1) == 0x75 && ishexstr(s+i+2)) {
+ for (unistr=utf8code16tostr(s+i+2); *unistr; j++, unistr++) {
+ *(s+j) = *unistr;
+ }
+ i += 6;
+ } else {
+ for(skip = utf8skip(s+i)+1; skip > 0; skip--) {
+ *(s+j) = *(s+i);
+ i++; j++;
+ }
+ }
+ }
+ *(s+j) = *(s+i);
+}
+
+
+/* Replace equal length substrings in s */
+char *streqrep(char *s, char *oldstring, char *newstring) {
+ char *ptr;
+ int len;
+ len = strlen(oldstring);
+
+ while ((ptr = strstr(s, oldstring)) != NULL) {
+ memcpy(ptr, newstring, len);
+ }
+ return(s);
+}
+
+int ishexstr (char *str) {
+ int i;
+ for (i=0; i<4; i++) {
+ if ((*(str+i) > 0x2f && *(str+i) < 0x3a) || (*(str+i) > 0x40 && *(str+i) < 0x47) || (*(str+i) > 0x60 && *(str+i) < 0x67))
+ continue;
+ return 0;
+ }
+ return 1;
+}
+int utf8strlen(char *str) {
+ int i,j, len;
+ len = strlen(str);
+ for (i=0, j=0; *(str+i) != '\0' && i < len;j++ ) {
+ i = i + utf8skip(str+i) + 1;
+ }
+ return j;
+}
+
+/* Checks if the next character in the string is a combining character */
+/* according to Unicode 7.0 */
+/* i.e. codepoints 0300-036F Combining Diacritical Marks */
+/* 1AB0-1ABE Combining Diacritical Marks Extended */
+/* 1DC0-1DFF Combining Diacritical Marks Supplement */
+/* 20D0-20F0 Combining Diacritical Marks for Symbols */
+/* FE20-FE2D Combining Half Marks */
+/* Returns number of bytes of char. representation, or 0 if not combining */
+
+int utf8iscombining(unsigned char *s) {
+ if (*s == '\0' || *(s+1) == '\0')
+ return 0;
+ if (!(*s == 0xcc || *s == 0xcd || *s == 0xe1 || *s == 0xe2 || *s == 0xef))
+ return 0;
+ /* 0300-036F */
+ if (*s == 0xcc && *(s+1) >= 0x80 && *(s+1) <= 0xbf)
+ return 2;
+ if (*s == 0xcd && *(s+1) >= 0x80 && *(s+1) <= 0xaf)
+ return 2;
+ if (*(s+2) == '\0')
+ return 0;
+ /* 1AB0-1ABE */
+ if (*s == 0xe1 && *(s+1) == 0xaa && *(s+2) >= 0xb0 && *(s+2) <= 0xbe)
+ return 3;
+ /* 1DC0-1DFF */
+ if (*s == 0xe1 && *(s+1) == 0xb7 && *(s+2) >= 0x80 && *(s+2) <= 0xbf)
+ return 3;
+ /* 20D0-20F0 */
+ if (*s == 0xe2 && *(s+1) == 0x83 && *(s+2) >= 0x90 && *(s+2) <= 0xb0)
+ return 3;
+ /* FE20-FE2D */
+ if (*s == 0xef && *(s+1) == 0xb8 && *(s+2) >= 0xa0 && *(s+2) <= 0xad)
+ return 3;
+ return 0;
+}
+
+int utf8skip(char *str) {
+ unsigned char s;
+
+ s = (unsigned char)(unsigned int) (*str);
+ if (s < 0x80)
+ return 0;
+ if ((s & 0xe0) == 0xc0) {
+ return 1;
+ }
+ if ((s & 0xf0) == 0xe0) {
+ return 2;
+ }
+ if ((s & 0xf8) == 0xf0) {
+ return 3;
+ }
+ return -1;
+}
+
+unsigned char *utf8code16tostr(char *str) {
+ int codepoint;
+ codepoint = (hexstrtoint(str) << 8) + hexstrtoint(str+2);
+ return(int2utf8str(codepoint));
+}
+
+unsigned char *int2utf8str(int codepoint) {
+ unsigned char *value;
+ value = xxmalloc(sizeof(unsigned char)*5);
+
+ if (codepoint < 0x80) {
+ *(value) = (unsigned char)(codepoint);
+ *(value+1) = 0;
+ return(value);
+ } else if (codepoint < 0x800) {
+ *(value) = (0xc0 | (unsigned char)(codepoint >> 6));
+ *(value+1) = (0x80 | (unsigned char)(codepoint & 0x3f));
+ *(value+2) = 0;
+ return(value);
+ } else if (codepoint < 0x10000) {
+ *(value) = (0xe0 | (unsigned char)(codepoint >> 12));
+ *(value+1) = (0x80 | (unsigned char)((codepoint >> 6) & 0x3f));
+ *(value+2) = (0x80 | (unsigned char)(codepoint & 0x3f));
+ *(value+3) = 0;
+ return(value);
+ } else {
+ return (0);
+ }
+}
+
+int hexstrtoint(char *str) {
+ int hex;
+
+ if (*str > 0x60) {
+ hex = (*str - 0x57) << 4;
+ } else if (*str > 0x40) {
+ hex = (*str - 0x37) << 4;
+ } else {
+ hex = (*str - 0x30) << 4;
+ }
+ if (*(str+1) > 0x60) {
+ hex += (*(str+1) - 0x57);
+ } else if (*(str+1) > 0x40) {
+ hex += (*(str+1) - 0x37);
+ } else {
+ hex += (*(str+1) - 0x30);
+ }
+ return hex;
+}
--
Alioth's /usr/local/bin/git-commit-notice on /srv/git.debian.org/git/debian-science/packages/foma.git
More information about the debian-science-commits
mailing list