------------------------------------------------------------------------------

                  What's new in each version of Interchange
                       (since the version 5.12 branch)

          See UPGRADE document for a list of incompatible changes.

------------------------------------------------------------------------------


Interchange 5.12 not released yet

Interchange now requires Perl 5.16.3 or newer.

Please note that the official Interchange project site has moved from
icdevgroup.org to:
https://www.interchangecommerce.org/

The canonical Git repository is now hosted on GitHub at:
https://github.com/interchange/interchange


Installation
------------

* Assume that if a source code control repository is present in the
  installation directory, we don't need to do the archaic .~* file copies.


Core
----

* Updated RobotUA handling.

* Improved UTF-8 handling and bugfixes.

* Improve image uploading via the table editor.

* Add support for [all-anchor] token in [more-list] template.

* Improve [image] extension handling to search for lower/uppercase versions
  of file extensions.

* Add date-change "common" filter parameter to output date+time without "T"
  separator.

* Add bcrypt filter.

* Allow a dash in the middle of a varname (and to end one).

* Allow changable varname regex for profile checks via
  Limit  profile_check_varname_regex   \b\w[-+.\w]*

* Fix bug where only one address was possible for DebugHost. Now accepts a
  range of IP addresses.

* Disallow comma character within email addresses in email and email_only
  profile checks.

* Add "default" option on CGI tag. Negligible impact on performance, as
  branch only followed on blank/zero value.

* Fix bug whereby certain LARGE or HUGE tables could have large searches
  started by hitting return with the cursor in the entry box.

* Fix bug in hash version of [item-tag-*].

* Reprioritize (later) little-used loop tag scratchd.

* Refactor cookie-setting routine to avoid double ; and extra string copying
  and use canonical capitalization.

* Teach pragma set_httponly to apply only to certain cookies.

  E.g. for the default session cookie only:

  Pragma set_httponly=MV_SESSION_ID

  Or for that plus another arbitrary cookie "cart":

  Pragma set_httponly=MV_SESSION_ID;cart

  The old behavior of setting HttpOnly for all cookies still works as before, e.g.:

  Pragma set_httponly

* Add CounterDir configuration to allow counters to be defined by default in
  some place different than VendRoot.

* Extend PerlAlwaysGlobal to encompass [loop-calc].

* Fixed a bug in the [area] tag when a full URL is passed as the href.

* Pass through from [bounce] URL-generation options such as form, secure,
  match_secure, etc.

* Add new AlwaysSecureGlob directive. It's not possible to enumerate all the
  admin URLs or ActionMaps that should be generated secure-only, so this new
  directive allows matching.

* Fix month and year adjustment wrapping issue in Vend::Util::adjust_time().

* Add [adjust-href] tag. Reads HTML passed to it, finds <a href> links
  and adjusts ones that don't begin with an absolute path to
  Interchange URLs. Normally done by setting "Pragma adjust_href" in
  catalog.cfg (or [pragma adjust_href] at the top of the page).

* Add code to Server.pm to adjust <a href="link"> tags when "Pragma adjust_href"
  is in force. Uses above tag.

* Cleanup of quite a few Perl module dependencies, removal of vestiges of
  old version control systems from source files.

* Improved integration of CI tools for development.

* Add ability to return after Preload and Autoload routines. Setting
  $Vend::PreloadReturn or $Vend::AutoloadReturn (i.e. they are defined)
  means Vend::Dispatch::dispatch() will return that status. This allows
  features like QueryCache and others that are hooked via Preload or
  Autoload.

* Add skeleton module Vend::InDev, which does nothing but set a single
  global variable if the file _indev is present in $Global::VendRoot.
  This allows interchange.cfg bifurcation, useful when maintain a SCCS
  repository.

* Multiple cleanups and bugfixes to handle more modern versions of Perl.

* Add x_accel_expires pragma to set X-Accel-Expires header for nginx proxy caching.

* Remove non-header from BounceReferrals HTTP response headers, which
  breaks output with newer Apache versions.

* Support recipient addresses in CC and BCC when using Net::SMTP.

* Fix Vend::Util::timecard_stamp() to support a passed timestamp.

* Make Interchange reopen debug.log/STDERR when receiving HUP signal.

* Fix race conditions in timed-build output.

* Added support for CIDR notation (for both IPv4 & IPv6) in TrustProxy and
  RobotIP directives.

* Normalize IPv6 addresses.

* Fix bug with mixed plaintext/crypted passwords on promotion.

* Quell error message for harmless empty JSON request body.

* Allow custom headers in [get-url].

* Add more unit tests.

* Make HTTP response codes more relevant.

* Improvements to [child-process] usertag.

* Allow setting cookies, even if no session is created.

* Add new global directive LogTimeFormat which allows customizing the prefix of
  log lines from the default [%d/%B/%Y:%H:%M:%S %z] using strftime(3) format.

* Add new SMTPConfig hash directive for Net::SMTP configuration to support
  SMTP AUTH user & password, alternate server port, SSL, and timeout options.

* New usertag [config] to pull catalog config params, or with 2nd-arg flag
  global config params. Supports dot-syntax for pulling from complex config
  params.

* New Pragma set_samesite to control SameSite settings on cookies.

  To set all cookies Strict:

  Pragma set_samesite=Strict

  To set only foo to Lax:

  Pragma set_samesite=foo=Lax

  To add bar as Strict:

  Pragma set_samesite=foo=Lax;bar=Strict

  And add baz as None:

  Pragma set_samesite=foo=Lax;bar=Strict;baz=None

  If None is used, that forces Secure on those cookies as well.

* Redis sessions and transaction locks for RDBMS sessions

  To use Redis for sessions, in catalog.cfg:
  - Set SessionType to Redis
  - Set Redis server address (with port) in SessionDB

  If you're also using MoreDB, in catalog.cfg:
  - New config param, MoreDBTable, to point to the RDBMS where you have the
    table to hold more lists.
  - In strap, the table, sessions, is created with the correct schema.
  - Without MoreDBTable, MoreDB falls back to using SessionDB, same as
    currently. But you must use MoreDBTable with Redis sessions.

  Note: Vend::SessionRedis maintains a fixed, 30-minute value applied to the
  session expiration for any new session. Bots clog up the sessions with their
  "hit and run" action. By limiting the first hit to a relatively small amount
  of time, it keeps down the buildup of session records that will never be used
  again. For human users, 30 minutes gives them ample time to make their first
  request with the established session, at which point the session expiration is
  extended to the duration configured with the SessionExpire directive.

Payments
--------

* Add BIN-2 MasterCard number support to Vend::Order, catalog templates,
  admin, and Vend::Payment modules which require it.

* Include b_phone and phone in payment information.

* Remove defunct payment services: Vend::Payment::ECHO and Vend::Payment::Skipjack.

* Updated Vend::Payment::PayPal to latest available version.

* New Braintree payment module.

* Add Gateway Log feature for logging full details of payment gateway requests/
  responses. (See extended section about the Gateway Log later in this document.)


Database
--------

* Support newer versions of MySQL.

* Prevent key-only inserts for new rows in [import-fields] tag.

* Allow admin user to export specific tables using [backup-database] despite
  current NoExportExternal setting.

* Add ability to have a NO_UPDATE field in SQL tables, which field will
  not affect a TIMESTAMP field (in MySQL or Postgres, at least).

  Intended to allow mod_time in user logins to be updated without changing
  a timestamp field. May have other uses, but is only honored at this
  time in the set_field() method, so has limited applicability.

  Assuming one has a timestamp field in the userdb table called
  "update_date", this is effected by changing the configuration as:

    Database userdb  TIMESTAMP_FIELD  update_date
    Database userdb  NO_UPDATE        mod_time

* Fix bugs with MySQL and PostgreSQL handling of QUOTE_IDENTIFIERS.

* Update default field widths for userdb, transactions, and orderlines.

* Improve error handling/messaging when no access to database in Safe.

* Support MySQL's utf8mb4 character set encoding, to be able to store all
  UTF-8 characters.


UserDB
------

* Add valref and scratchref options to UserDB, allowing the admin user to put
  their stuff in someplace besides $Values. This would allow you to easily set
  $Values to customer or affiliate settings, without impacting the admin user.

  If you do:

    UserDB  ui  valref   user_record
    UserTag uvalue Alias  data base=session key=

  you will then be able to reference the Admin "values" with

    [uvalue name]

  The [if-mm] tag probably needs to be adjusted for this to work
  in the admin in real life.

* Check for admin status earlier so it's available to postlogin_action.

* Add support for prelogout_action to UserDB logout.

* Add "fallback_login" option, to be used with "indirect_login". If
  indirect fails, it will fallback to the primary key (by default,
  username). This could allow users to login with email (indirect_login
  = usernick), but still support login via username if they opted to use
  their username instead.

* Add "promote_admin" option, to be used with "scratch". The option
  is set to the value of a field name, which also has to be a "scratch"
  value. If that is true, and the value of the field is true, the user
  will be promoted to $Vend::admin. They will not be $Vend::super. Note
  that this does not gain access to the classic Interchange UI without
  modification, since the login_table will not be admin (the test which is
  used in most cases).

* Fix bug causing passwords to be blanked in the database when promoting
  to e.g. bcrypt hashing but required modules are missing. This caused the
  user to be unable to log in thereafter until a new password was set, but
  did not allow any unauthorized access. (GitHub pull request #90)


Admin
-----

* Fix XSS injection vulnerabilities in help and quicklinks (CVE-2020-12685).

* Respect custom BACKUP_DIRECTORY in admin table downloads. This matches
  behavior expected by [export-database].

* Fix bug which blew away user's yes_tables and yes_functions setting when
  editing name/superuser status.
`
* Fix table editor composite key problems, both creating new and editing
  existing rows.

* Exclude canceled orders from reports.

* Remove support for ancient browsers and unused variables.

* Remove old help content.

* Move access table into SQL to avoid locking between users. (GitHub pull
  request #89)


Strap Demo Catalog
------------------

* Updated the versions of jQuery, Google Analytics, and Bootstrap used.

* Turn off UserDB ignore_case by default.

* Prevent browser autocomplete problems.

* Remove some legacy cart components.

* Formatting fixes on checkout pages.

* Fix broken random password number range; consistently make random
  instead of using zip or phone.

* Lengthen ip_addr database fields to support IPv6.

* Remove the option for session IDs in URLs, so cookies are always required.

* Add base configuration and settings to delegate [salestax] to 3rd-party tax
  APIs generally, and use of Tax Jar specifically.

* Support easy use of SQLite in makecat.


Other
-----

* Updated te utility with several new features.

* Quickbooks integration: Fix typos so that removing stray trailing
  whitespace works.

* Added tools for viewing/manipulating existing Storage sessions.

* Removed old mod_perl artifacts.

* Added support for {NEWLINE} token in [error] templates, by René Hertell
  from GitHub issue #40.

* Removed Vend::Email module that was added but never integrated
  (GitHub issue #58).

* Taught [email-raw] to properly handle Unicode via the MV_EMAIL_CHARSET
  variable.

* Removed vestigial Windows support and defunct Irix support.


Gateway Log
-----------

Vend::Payment::GatewayLog - Basic package and methods for enabling full
transaction logging in any of the gateways within the Vend::Payment::*
namespace.

Gateway logging is inactive by default. It can be explicitly enabled by
triggering the "gwl_enabled" option in any of the usual ways:

* As an option directly through the [charge] tag. E.g.,
  [charge route=authorizenet gwl_enabled=1 ...]

* As an option defined in a payment route. E.g.,

  Route  authorizenet  id           "__MV_PAYMENT_ID__"
  Route  authorizenet  gwl_enabled  1
  ...

* Globally for all payment modules where gateway logging support has been
  added to the module, via MV_PAYMENT_* mechanism. E.g.,

  Variable  MV_PAYMENT_GWL_ENABLED 1

Note at this time, only a few payment modules have been fitted with gateway
logging support. It is assumed that the developer of each module, who is
familiar with the request/response structure of the specific API, will be
ideally suited to add gateway logging to the remaining modules. Those
currently supporting it are:

* AuthorizeNet
* Braintree
* CyberSource
* PayflowPro
* PaypalExpress (for dorequest API activity only)

Developers responsible for other payment gateways are encouraged to follow the
examples of the above payment modules and outfit remaining payment gateways
with their own gateway-logging hooks.

The format and fields of the gateway_log table as defined in the strap demo
should be used for maximal interoperability. The table name can be any name
you like, but because every payment module independently constructs the code
to populate the table, changing the field data types or names, and/or
adding/removing fields, will likely cause malfunction for the existing
modules' logging configurations.

There are 3 settings a catalog manager can control when enabling gateway
logging:

* Enabled (discussed above)
  Boolean to indicate that actual logging should be performed. Default
  is false; thus logging must be explicitly requested. Can be set with
  Route param or [charge] options "gwl_enabled", or globally with
  MV_PAYMENT_GWL_ENABLED in catalog.cfg.

* LogTable
  Name of table to which logging should be directed. Default is
  gateway_log. Can be set with Route param or [charge] option "gwl_table",
  or globally with MV_PAYMENT_GWL_TABLE in catalog.cfg.

* Source
  Maps to the request_source field in the log table. Value is most
  meaningful in a distributed environment, where multiple servers
  running the Interchange application may be handling requests behind a
  load balancer. Default value obtained from `hostname -s`. Can be set
  with Route param or [charge] option "gwl_source", or globally with
  MV_PAYMENT_GWL_SOURCE in catalog.cfg.


3rd Party Tax Support
---------------------

Vend::Tax - alternative salestax framework to allow catalogs to leverage
3rd-party APIs for tax calculation, estimates, and reporting.

Vend::Tax defines 3 new tags to support each of the above identified
functions:

* [tax-lookup] - returns calculated tax amount determined by specific
  3rd-party provider. Tax may be estimated or live lookup, depending on
  settings. Data required to calculate tax will be provider dependent.

* [load-tax-averages] - requests and stores tax averages for running in
  estimate mode, for providers that support it. Stores estimates by default in
  "tax_averages" table. Further, allows for local tracking of jurisdictions
  with nexus, which can be used by live lookups to determine if a particular
  lookup can be skipped entirely. See load_tax_averages Job and "tax_averages"
  table definition in strap demo.

* [send-tax-transaction] - report to provider the resulting tax transaction
  for a given order, for providers that support it. By default, operates based
  on transactions.tax_sent field. 0/empty indicates transaction not reported.
  1 indicates transaction successfully reported. -1 indicates an error
  attempting to report transaction, requiring manual intervention. See
  send_tax_transactions Job in strap demo.

Conceptually, Vend::Tax is patterned off of Vend::Payment for payment
transactions. It provides the defined interface that Interchange will use,
via the 3 tags shown above, but must delegate to vendor-specific
implementations through tax gateway modules.

Currently, only Tax Jar is supported - see Vend::Tax::TaxJar. But Vend::Tax
was designed to facilitate and encourage the development of any 3rd-party
providers through the creation of new Vend::Tax::Service modules. A review of
Vend::Tax::TaxJar should be instructive for the expected interface with the 3
usertags.

Like the payment modules, in order to make a particular tax service available,
it must be "Required" from interchange.cfg. E.g.:

 Require module Vend::Tax::TaxJar

See strap demo for example on integrating 3rd-party tax into your catalog via
the SalesTax configuration directive.


(end)
