Bug#379107: Reasoning behind 'encoding'/'termencoding' differences

[Kurt Roeckx]

On Thu, Apr 03, 2008 at 12:27:01AM -0400, James Vega wrote:
> On Sat, Dec 22, 2007 at 02:27:00PM +0100, Kurt Roeckx wrote:
> > On Sat, Dec 22, 2007 at 01:36:48AM -0500, James Vega wrote:
> > > After looking around in the help, the reasoning behind the current
> > > behavior seems to be described in ":help encoding-table" and in general,
> > > using the following will get the behavior you want:
> > > 
> > >   :let &termencoding = &encoding
> > >   :set encoding=utf-8
> > > 
> > > There are various caveats explained in that help section though.  If you
> > > feel the description and workaround are sufficient, please close the
> > > bug.  Otherwise, I would be interested in hearing how this doesn't meet
> > > your requirements.
> > 
> > So, what I would like is that atleast the
> > ":let &termencoding = &encoding" is being done by default.  This seems
> > to do exactly what I want: base the termencoding on the LC_CTYPE.
> 
> That is exactly what Vim does by default.  The problem is that you're
> specifically telling Vim to ignore the environment for 'encoding' and
> therefore need to tell Vim to use a different, possibly incompatible,
> encoding when displaying the buffer and interpreting input.
> 
> Vim takes the safe choice and always bases 'termencoding' on 'encoding'
> unless you tell it to do otherwise.  Your suggestion would be to remove
> the middle-man ('encoding') and simply have 'termencoding' directly use
> the the environment?

Having termencoding directly use the environment would be the "safe
choice" to me.  That is what that environment variable is for.

To restate my problem:
- I want to be able to edit files that are not written in whatever my
  environment says and that it properly interpretes and shows that file
  to me.  This does not happen with the default values.
  
  So that means that I want I want fileencodings to contain something so
  that it looks what encoding the file is so it can properly set
  fileencoding.

  With the settings I use now, and what I really want, is:
  ucs-bom,utf-8,default,latin1

  The default value without changing settings is: ucs-bom

  For this to work, it needs to be able to convert from fileencoding
  to termencoding.  For this to work, it needs to internally work in an
  "encoding" that supports both.  The most logical choise for that is
  unicode / utf-8.

- I would like that (new) files I create to be stored as utf-8, even
  though they all contain characters written in latin1.

So, the settings I would like to have so that everything works properly
is:
termencoding=latin1
encoding=utf-8
fileencodings=ucs-bom,utf-8,default,latin1

To get this, I now use:
:let &termencoding = &encoding
set encoding=utf-8

The termencoding should be based on the LCTYPE, and what "locale charmap"
or nl_langinfo(CODESET) returns.  I see no reason why you ever want this
to be something else.

I need to set the encoding to utf-8 so that I can properly edit files
that are written in utf-8 either by me or someone else.

I only had one problems using those settings so far, and that was an
interaction between mutt and vim not agreeing on the encoding of the
file.  For this reason I also have to use:
au FileType mail setlocal fenc=latin1


Kurt