Bug#574029: jh_manifest utf8 errors

Sat May 1 15:16:43 UTC 2010

Hi,

OK, did some progress but it's very weird...

I've patched jh_manifest to get some debug information (the last change 
in the attached patch, the others are just to get rid of the other 
warning about deprecation).

Here is the output (with LANG unset):

$ /tmp/jh_manifest -v -plibknopflerfish-osgi-framework-java
Updating manifest in 
/home/ericl/ALIOTH/build-area/knopflerfish-osgi-2.3.3/debian/libknopflerfish-osgi-framework-java/usr/share/java/framework-2.3.3.jar
DEBUG: sec[i] = 1.0
DEBUG: sec[i] = Apache Ant 1.7.1
DEBUG: sec[i] = 14.0-b16 (Sun Microsystems Inc.)
DEBUG: sec[i] = org.knopflerfish.framework.Main
DEBUG: sec[i] = OSGi Framework API
DEBUG: sec[i] = 1.3
DEBUG: sec[i] = Open Service Gateway initiative
DEBUG: sec[i] = Knopflerfish OSGi Framework
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = frameworkbundle
DEBUG: sec[i] = Knopflerfish OSGi framework system bundle
DEBUG: sec[i] = org.osgi.framework;specification-version=1.3,org.osgi.
  service.packageadmin;specification-version=1.2,org.osgi.service.permi
  ssionadmin;specification-version=1.2,org.osgi.util.tracker;specificat
  ion-version=1.3.1,org.osgi.service.startlevel;specification-version=1
  .0,org.osgi.service.condpermadmin;specification-version=1.0
DEBUG: sec[i] = http://www.knopflerfish.org
aEBUG: sec[i] = Build Sat May 1 2010, 13:58:36
Updating manifest in 
/home/ericl/ALIOTH/build-area/knopflerfish-osgi-2.3.3/debian/libknopflerfish-osgi-framework-java/usr/share/java/framework-2.3.3.jar
DEBUG: sec[i] = 1.0
DEBUG: sec[i] = Apache Ant 1.7.1
DEBUG: sec[i] = 14.0-b16 (Sun Microsystems Inc.)
DEBUG: sec[i] = org.knopflerfish.framework.Main
DEBUG: sec[i] = OSGi Framework API
DEBUG: sec[i] = 1.3
DEBUG: sec[i] = Open Service Gateway initiative
DEBUG: sec[i] = Knopflerfish OSGi Framework
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = frameworkbundle
DEBUG: sec[i] = Knopflerfish OSGi framework system bundle
DEBUG: sec[i] = org.osgi.framework;specification-version=1.3,org.osgi.
  service.packageadmin;specification-version=1.2,org.osgi.service.permi
  ssionadmin;specification-version=1.2,org.osgi.util.tracker;specificat
  ion-version=1.3.1,org.osgi.service.startlevel;specification-version=1
  .0,org.osgi.service.condpermadmin;specification-version=1.0
DEBUG: sec[i] = http://www.knopflerfish.org
3EBUG: sec[i] = Build Sat May 1 2010, 13:58:36

First weirdness is that jh_manifest updates twice the manifest of the 
same jar file.

Second weirdness is that the first letter of "DEBUG" for the last line 
with the date is overwritten with a random character (the a and the 3, 
always the same in all my tests).

But, as you can see, jh_manifest didn't obviously failed, and it 
actually is quite random, sometimes it works, sometimes it doesn't, 
especially if I go out and back into the chroot, it generally switches 
behavior. Also weird.

And, 4th weirdness, I can generally bring it to crash (or not crash) by 
changing the environment, LANG appears to be only a variable like any 
other, e.g.
$ X=Y /tmp/jh_manifest -v -plibknopflerfish-osgi-framework-java
Updating manifest in 
/home/ericl/ALIOTH/build-area/knopflerfish-osgi-2.3.3/debian/libknopflerfish-osgi-framework-java/usr/share/java/framework-2.3.3.jar
DEBUG: sec[i] = 1.0 

DEBUG: sec[i] = Apache Ant 1.7.1 

DEBUG: sec[i] = 14.0-b16 (Sun Microsystems Inc.)
DEBUG: sec[i] = org.knopflerfish.framework.Main
DEBUG: sec[i] = OSGi Framework API
DEBUG: sec[i] = 1.3
DEBUG: sec[i] = Open Service Gateway initiative
DEBUG: sec[i] = Knopflerfish OSGi Framework
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = frameworkbundle
DEBUG: sec[i] = Knopflerfish OSGi framework system bundle
DEBUG: sec[i] = org.osgi.framework;specification-version=1.3,org.osgi.
  service.packageadmin;specification-version=1.2,org.osgi.service.permi
  ssionadmin;specification-version=1.2,org.osgi.util.tracker;specificat
  ion-version=1.3.1,org.osgi.service.startlevel;specification-version=1
  .0,org.osgi.service.condpermadmin;specification-version=1.0
DEBUG: sec[i] = http://www.knopflerfish.org
aEBUG: sec[i] = Build Sat May 1 2010, 13:58:36
Updating manifest in 
/home/ericl/ALIOTH/build-area/knopflerfish-osgi-2.3.3/debian/libknopflerfish-osgi-framework-java/usr/share/java/framework-2.3.3.jar
DEBUG: sec[i] = 1.0
DEBUG: sec[i] = Apache Ant 1.7.1
DEBUG: sec[i] = 14.0-b16 (Sun Microsystems Inc.)
DEBUG: sec[i] = org.knopflerfish.framework.Main
DEBUG: sec[i] = OSGi Framework API
DEBUG: sec[i] = 1.3
DEBUG: sec[i] = Open Service Gateway initiative
DEBUG: sec[i] = Knopflerfish OSGi Framework
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = Knopflerfish
DEBUG: sec[i] = 4.1.10
DEBUG: sec[i] = frameworkbundle
DEBUG: sec[i] = Knopflerfish OSGi framework system bundle
DEBUG: sec[i] = org.osgi.framework;specification-version=1.3,org.osgi.
  service.packageadmin;specification-version=1.2,org.osgi.service.permi
  ssionadmin;specification-version=1.2,org.osgi.util.tracker;specificat
  ion-version=1.3.1,org.osgi.service.startlevel;specification-version=1
  .0,org.osgi.service.condpermadmin;specification-version=1.0
DEBUG: sec[i] = http://www.knopflerfish.org
Traceback (most recent call last):
   File "/tmp/jh_manifest", line 265, in <module>
     updatejar(j, tempmanifest, args)
   File "/tmp/jh_manifest", line 78, in updatejar
     manifest = parseManifest(tempdir + "/META-INF/MANIFEST.MF", None, 
manifest)
   File "/tmp/jh_manifest", line 164, in parseManifest
     print "DEBUG: sec[i] = " + sec[i]
   File "/usr/lib/pymodules/python2.5/debian/deb822.py", line 179, in 
__getitem__
     value = value.decode(self.encoding)
   File "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode
     return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x81 in position 31: 
unexpected code byte

Note that the last line has 30 characters, so 31 is just after, but 
there is no 0x81 character in the manifest file:

$ hexdump -C META-INF/MANIFEST.MF | tail
00000340  73 67 69 2e 73 65 72 76  69 63 65 2e 63 6f 6e 64 
|sgi.service.cond|
00000350  70 65 72 6d 61 64 6d 69  6e 3b 73 70 65 63 69 66 
|permadmin;specif|
00000360  69 63 61 74 69 6f 6e 2d  76 65 72 73 69 6f 6e 3d 
|ication-version=|
00000370  31 2e 30 0d 0a 42 75 6e  64 6c 65 2d 44 6f 63 55 
|1.0..Bundle-DocU|
00000380  52 4c 3a 20 68 74 74 70  3a 2f 2f 77 77 77 2e 6b  |RL: 
http://www.k|
00000390  6e 6f 70 66 6c 65 72 66  69 73 68 2e 6f 72 67 0d 
|nopflerfish.org.|
000003a0  0a 42 75 69 6c 64 2d 44  61 74 65 3a 20 42 75 69 
|.Build-Date: Bui|
000003b0  6c 64 20 53 61 74 20 4d  61 79 20 31 20 32 30 31  |ld Sat May 
1 201|
000003c0  30 2c 20 31 33 3a 35 38  3a 33 36 0d 0a 0d 0a     |0, 
13:58:36....|
000003cf

Bottom line, I suspect that there is something very wrong in the 
underlying library because it's just by *reading* sec[i] that 
jh_manifest stumbles, so that a wild guess of mine would be that the 
error actually happens while filling the variable with "for sec in 
debian.deb822.Deb822.iter_paragraphs(f):" but then I can barely read 
python so I have to give up here.

Hope this helps you further fix the problem.

Very last thing: the workaround with override_jh_manifest doesn't seem 
to work for me, jh_manifest gets called and happily fails...

Eric
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jh_manifest.diff
Type: text/x-diff
Size: 1276 bytes
Desc: not available
URL: <http://lists.alioth.debian.org/pipermail/pkg-java-maintainers/attachments/20100501/89e8a4a7/attachment-0001.diff>