mod_archive_odbc release

Tagged:

This article is about mod_archive_odbc: ejabberd implementation of server-side instant messaging history support, also known as XEP-136.

Update: development of this module has stopped, all further development is done in mod_archive2 project.

Development history

Original version of module for XEP-136 support in ejabberd, mod_archive was written by Olivier Goffart in 2006, see his post.

Without this original great work I do not think I would even start working on this subject at all, as I didn’t know Erlang and had no idea on where to start from.

When trying to experiment with this original mod_archive implementation I soon discovered some bugs in RSM handling, as well as missing features here and there.

I’ve performed some fixing, added replication support and submitted patch in ejabberd bugtracking system.

However, the patch was not applied as at this time there was already another implementation of mod_archive: mod_archive_sql, which is basically the port of original, mnesia-based mod_archive to PostgreSQL RDBMS, and ejabberd developers didn’t want these versions to differ.

To be true, I find the approach of maintaining two separate versions of the same module that differ only by storage engine pretty ridiculous, and for sure I wasn’t going to participate in that “road to hell” - I worked once for one big project that had very high rate of duplicate code (hello to those my ex co-workers who’re reading this! , and I know how bad it is …

Therefore, I’ve stopped at that time with further mod_archive fixes and switched to the other task - writing client library for XEP-136 support, to make it easy for clients to adopt this protocol. The library is subject for separate post, but what is important is that when I started to test it against server part, it quickly appeared that there were some severe limitations in mod_archive architecture, which were also inherited by mod_archive_sql version:

  • RSM filtering, which is basically used to limit the number of queries and to page through results, was applied after fetching items satisfying the main request.
  • When answering requests related to collections, they were fully fetched from the database, including all messages they contained.

These two points effectively mean that if client performs the request as simple as “show last 10 collections” the entire database is picked up in memory, and only after that filtering is performed to select those 10 collections the user asked for.

With about 3,000 collections recorded in mnesia database ejabberd easily took about 1Gb of RAM answering these queries, the speed was quite bad also - I do not think it would be useable at all with even larger database.

Just to put it to the right scale: currently I have the messaging archive with about 28,000 collections and around 290,000 messages, I cannot imagine processing these volumes of data with architecture explained above.

Therefore I had to return to the task of mod_archive development again.

First I just wanted to further patch original, mnesia-based mod_archive version to make it more efficient - but soon I discovered that mnesia just has not enough power to perform the tasks I needed. The most severe limiting factor was that I found no ways to efficiently limit queries size and page through results. Switching to SQL-based version seemd to be the only viable solution.

So I took mnesia-based mod_archive version, mod_archive_sql and tried to produce resulting version which would have the correct architecture & implementation to be efficient enough.

This appeared to be not that easy task - in the end it looks like I’ve rewritten most important parts of the module almost completely.

However, the current implementation I have seems to be quite effective, it easily proceeds the volumes of data I mentioned above and also much more conformant to XEP-136 standard.

I will submit this version to ejabberd bug-tracking system once more - I hope it can be included now into ejabberd trunk, replacing both mod_archive and mod_archive_sql, as I believe there’s no point in further development of mnesia-based mod_archive.

Current status

Current version is called mod_archive_odbc, as it’s designed (in theory) to work with any ODBC-compliant RDBMS. On practice there are quite some incompatibilities between RDBMS implementations, see below for details.

To the best of my knowledge and understanding of the standard, this version is fully conformant to XEP-136 except of the following:

  • Encryption is not supported.
  • Setting “auto” preference is persistent, not per-session as it should be according to XEP-136 (thanks to Olivier Goffart for spotting this!)
  • When auto-archiving is enforced by module options, warning message, that is required according to XEP-136, is not sent to legacy clients.
  • Remove collection” command with attribute “open=true” is not supported - I do not think it’s very important to have at current stage, though.

All examples from XEP-136 standard, excluding those related to features mentioned above, seems to work correctly with current version.

Please note though that the standard is not yet finalized, so it’s still possible there will be some further changes in it.

Also there were revealed some problems in standard during implementation, I will submit the report to XMPP guys about them - hopefully, these still can be fixed before standard approval. mod_archive_odbc contains fixes to these problems - as these fixes do not break backward compatibility, it should be completely safe to have them there.

If you run mod_archive_odbc and encounter any standard-breaking behavior, or just any bug with it - I’d like to hear about it, please consider either contacting me or posting the comment to this article.

Please note that this module is still in experimental state! You’re strongly adviced to avoid using it on production servers, or at least make sure that frequent database backups are performed.

Change History

Revision 476

  • Fixed error when requests without RSM tag were not processed, by Olivier Goffart.
  • Added auto-archiving for groupchats.
  • Fixes to support native MySQL driver.

Revision 465

Replaced string:to_lower by jlib:tolower everywhere in sources, as it appeated that string:to_lower was introduced as late as in erlang R11-B4 so not everybody have it.

Revision 434

Fixed bug with crashes on empty RSM responses.

Version 1.0.1

Fixed serious bug which might cause wiping out extra collections in the case user uploads preferences which specify expire interval for full or bare JID, but there is no prefs entry for domain of this JID - expire interval of this JID was applied on the whole domain, possibly causing removal of unwanted collections.

All users are adviced to upgrade or, if upgrade is not possible, the workaround for version 1.0.0 is to set wipeout_interval option to infinity.

Upgrading instructions: DB schemas were not changed, so just copy new version of mod_archive_odbc.erl to ejabberd sources and recompile.

Version 1.0.0

Initial release.

Downloading and Installation

The most up-to-date version is located at ejabberd-modules SVN repository. The files you need are with mod_archive_odbc* prefix.

Just put mod_archive_odbc to ejabberd source tree before ejabberd compilation, then compile ejabberd as usual. Do not forget to turn on ODBC support using --enable-odbc.

mod_archive_odbc was tested with ejabberd SVN trunk and ejabberd-1.1.4 versions and works with both of them. However, for ejabberd-1.x versions the following needs to be done:

  1. Supporting module ejabberd_odbc_sup has to be started - if you use auth_odbc it is already started there, if not - you should start it explicitly from mod_archive_odbc, just uncomment lines “ejabberd-1.x compatibility code”. In ejabberd SVN this is done always automatically by ejabberd_rdbms, so you do not have to worry about it.
  2. Connection to RDBMS can be closed by timeout if not used for a long time, causing the module to fail. In ejabberd SVN version there’s odbc_keepalive_interval parameter which can be used to periodically issue dummy query to keep connection up, but there’s no such thing in ejabberd-1.x, so you’ll have to deal with it on your own either switching timeout off in RDBMS configuration, issuing periodically the command to mod_archive_odbc or in any other way you find suitable for your configuration.

Please note that odbcserver, which is part of ODBC support package for erlang, has limit 8001 bytes on the maximal size of column data returned - this may become a problem if you have large messages in your collections, as they will be truncated on retrieval from database.

You can use this patch for erlang-11.2.5 to lift the restriction up to 65535 bytes. You’ll need to recompile odbcserver, either separately on its own - or just as part of whole erlang compilation.

Configuration

The following steps need to be done to configure mod_archive_odbc to use ODBC database connection:

  1. Create database using either one of schema files provided with mod_archive_odbc or your own, making sure, of course, that it’s compatible to the schema mod_archive_odbc expects.
  2. Create DSN to your database: for Windows use Administrative Tools → Data Sources (ODBC), for *nixes - configs for your unixODBC installation. It’s a good idea to check at this point DSN works correctly before proceeding further, preferrably - using erlang + odbc, see erlang odbc docs for an example, or even better - using ejabberd ODBC wrapper, see section “Debugging the ODBC configuration”, using ejabberd_odbc:sql_query. For example, smth like ejabberd_odbc:sql_query("localhost", "select * from archive_messages").
  3. Specify {odbc_server, “Your ejabberd database DSN”} in ejabberd.cfg.
  4. Add mod_archive_odbc to the list of modules ejabberd loads, parameters are described below.

For native connections, such as native MySQL driver, after creating database you should just add the appropriate option to ejabberd.cfg specifying database to connect and that’s should be it, see ejabberd site articles for general discussion, as well as native MySQL driver tutorial.

Parameters mod_archive_odbc understands:

  • database_type - selects RDBMS type, currently “mysql”, “sqlite”, and “pgsql” are supported. This parameter should always be specified, all subsequent parameters are optional.
  • default_auto_save (true or false, defaults to false) - whether to perform auto-saving by default or not; please note that XEP-136 requires it to be off by default.
  • enforce_default_auto_save (true or false, defaults to false) - whether to enforce default_auto_save option by prohibiting users changing it individually in their preferences or not.
  • session_duration (time in seconds) - the timeout interval the server uses to decide that conversation is over when performing automated archiving.
  • default_expire (time in seconds or infinity atom, defaults to infinity) - all collections with their start time satisfying now() - start() >= default_expire will be removed, unless expire time is overridden by users in their preferences.
  • enforce_min_expire (time in seconds, defaults to 0) - minimal expire time users are allowed to set in their preferences, useful if you’d like to enforce some message saving policy. NOTE: this parameter is not currently checked in modification commands, such as “upload collection” or “remove collection”, so you cannot enforce the policy completely yet!
  • enforce_max_expire (time in seconds or infinity atom, defaults to infinity)
    • maximum expire time user is allowed to set, useful for limiting the amount of collections users may keep on server.
  • replication_expire (time in seconds or infinity atom, defaults to 31536000, which is one year) - all collections with their removal time satisfying now() - removal_time() >= replication_expire will be removed completely from the database, thus the server completely forgets they were present once.
  • wipeout_interval (time in seconds or infinity atom, defaults to 86400, which is one day) - interval between executions of expire routine.

Please note that according to XEP-136 only the following auto_save combinations are valid:

  1. default_auto_save = true, enforce_default_auto_save = true
  2. default_auto_save = false, enforce_default_auto_save = false

Implementation will happily work with any combination of these, though - for example, for personal ejabberd server, until all clients support XEP-136, combination default_auto_save = true, enforce_default_auto_save = false is quite logical, while for some public ejabberd server with lots of users and shortage of disk space default_auto_save = false, enforce_default_auto_save = true might be desirable, so that only manual uploads are supported.

Here’s the example configuration when using ODBC MySQL driver:

...

% Add to modules section:
{mod_archive_odbc, [{database_type, "mysql"},
                      {default_auto_save, true},
                      {enforce_default_auto_save, false},
                      {default_expire, infinity},
                      {enforce_min_expire, 0},
                      {enforce_max_expire, infinity},
                      {replication_expire, 31536000},
                      {session_duration, 1800},
                      {wipeout_interval, 86400}]},

...

% Add somewhere at top level, if you do not have it yet:
{odbc_server, "DSN=ejabberd;UID=ejabberd;PWD=passwd"}.

% Make sure the connection isn't closed,
% this option is supported in ejabberd-2.x only.
{odbc_keepalive_interval, 3600}.

...

For native MySQL driver replace “odbc_server” connection string by smth like this:

...

{odbc_server, {mysql, "localhost", "ejabberd", "ejabberd", "password"}}.

...

Migrating history

There’s no easy way to fully import history for all users from previous versions of mod_archive or other server-side storages to mod_archive_odbc - however, for XEP-136 servers implementations you can do it per user basis using wswutil by downloading history first to local file and then uploading it to server which runs mod_archive_odbc.

Please note, though, that previous versions of mod_archive had some of the features required by wswutil broken, so I cannot guarantee that downloading will succeed: I’ve tested this only with patched version of mnesia-based mod_archive. Also be warned that downloading can take large amounts of memory and be quite slow for the reasons explained above.

In wswutil distribution there are also some Perl scripts for history convertion to XEP-136 XML file format for the following clients:

  • Miranda (msg_export and dbeditorpp plugins should be used first to export history and database to text files)
  • Psi
  • vicq

However, these scripts are quite hacky and may require some finishing of resulting XML files before uploading.

I strongly suggest to test obtained XML files on some test server running mod_archive_odbc before uploading them on production one, as if it’s determined in the middle of uploading that XML is broken you’ll stay with half-way imported history and will have to find the way to remove all uploaded collections to start from scratch, which may be problematic if you have already some other collections stored in database which you’d like to keep.

Client support

So far nothing encouraging can be written here - I know no Jabber clients supporting current version of XEP-136. There’s JWChat which claims to support XEP-136, but it uses old version of specs so I’m almost sure it will not work with mod_archive_odbc.

One of the possibilities to get access to the history stored by mod_archive_odbc is to use Olivier’s Jabber archive web reader - it provides web interface to history on Jabber server that supports XEP-136, such as ejabberd + mod_archive_odbc.

Other RDBMS support

mod_archive_odbc utilizes ejabberd wrappers over ODBC - this means that in theory it should work with all RDBMS’es which have ODBC driver, as well as through native drivers to MySQL and PostgreSQL, as they are included in ejabberd and supported through the same interface as ODBC queries.

On practice, though, due to SQL standard being … well … just like any other standard, i.e. missing important features and not fully supported by any of the RDBMS in any of standard / RDBMS versions, there are some either non-standard or incompatible features used.

My tests covered the following setups:

  1. Windows XP + SQLite3 through ODBC driver.
  2. Linux/MySQL through ODBC driver.
  3. Linux/MySQL through native driver (only partially tested).

Therefore, for those configurations I can be pretty sure they’re working. There’s also support for PostgreSQL in sources, but it was never tested - so, most likely, it will not work out of the box, though should be easy to fix.

Update: it was reported that it works also with PostgreSQL, the appropriate schema file is now included in mod_archive_odbc, but I haven’t tried it myself.

Update2: it was reported that collections expiration is broken on PostgreSQL, see comments to this post for details.

For other RDBMS’es like MS SQL, Oracle, Firebird or whatever, you will need to read on to figure out what should be changed to support them.

This comparison of different SQL implementations, which lists incompatibilities between different RDBMS’es, helped me very much during development, if you’re thinking about adding support to mod_archive_odbc for other RDBMS’es - it can be useful for you also.

Below I list all the features I know in current implementation to behave not according to standard or that are likely to be a problem for new RDBMS support.

Almost all differences are localized to the appropriate case construction, so you just need to search the source for all database_type instances, and add your RDBMS into the appropriate case statement.

Triggers

Triggers are still new to some of RDBMS’es, so there may be some deviations here and there. In my tests they worked perfectly for SQLite3, but required quite some tuning for MySQL. Specifically, MySQL doesn’t support triggers updating the same table that invoked these triggers - therefore, for MySQL I had to put some actions directly in mod_archive_odbc code instead of triggers.

Depending on situation with your RDBMS you can consider either going SQLite3 direction, so just using triggers (preferred way) or use the same approach as for MySQL.

Paging through results and limiting result size

I have no clues how could that happen that this is not in the standard and implementations are incompatible with respect to this feature. To be more precise, there IS functionality in the standard that allows to implement this, albeit in quite awkward way, but it’s not supported by majority of RDBMS’es anyway, so there’s not that much sense in it.

Due to RDBMS’es selection used for my tests, I’ve used "OFFSET ... LIMIT ..." construction to acomplish the goal - it should be portable at least between SQLite, MySQL and PostgreSQL, and is quite easy to understand.

This one is important to fix for your RDBMS, as it provides the essential functionality, but it’s also one of the most difficult ones - if your RDBMS requires completely different way of query construction you may need to make quite some changes. They’re localized to generic query constructions functions, though, so at least you do not have to do this separately for collections, messages and changes queries.

Last Inserted Auto ID

When auto incremented column, used as primary key, is updated on new record insertion, I need in some places to know its value that was just inserted. Again, surprisingly there’s quite some deviation with that: almost every RDBMS has this function, but it’s called differently everywhere.

This one is easy - just find out the right way to do it for your RDBMS and insert it into the appropriate case.

In principle, you can even skip this completely, if this function is not available - implementation will just perform normal SELECT statement, that should also work in most cases, although being slightly slower and, theoretically, subject to problems if duplicate records are inserted - which should not be the case on practice, as last inserted id functionality is currently used for collections insertions only.

Strings escaping

The standard specifies that strings are specified as ‘string’, and one ’ symbol is escaped as ”, so the string can't should become 'can''t'. Besides that, \ symbol does not have special meaning according to the standard.

However, some RDBMS’es allow \ to be used for escaping, which creates the mess, as strings have to be escaped differently: for those that strictly follow the standard a\b'c should be escaped as 'a\b''c', while for others 'a\\\\b\'c'.

Again, this one is simple to handle: just add the case for your RDBMS in escape_str() function.

Multiple insert

For optimum performance it’s better to insert messages in database not one-by-one, but in bulk requests, this makes quite some difference - with single inserts uploading about 3000 collections into MySQL over local network takes more than an hour, while with bulk inserts - less than 10 minutes. The problem is that appropriate INSERT operation is specified as optional in SQL standard, and is not implemented by some RDBMS’es.

Just change case in store_messages() function to cover your RDBMS correctly.

Date/Time functions

For expiration support there’s a need to perform some calculations with date/time in query - again, everyone is different here.

Change case in get_expired_str() to cover your RDBMS, if you need expiration support.

Database creation and types

I tried to stick to very basic types in database schema to make it easier to port implementation to other RDBMS’es, as well as to work-around odbc problems in erlang implementation. Therefore, for example, BOOLEAN type is not used and is discouraged - use INTEGER or TINYINT instead. For timestamps everything that is able to store the date in format ‘YYYY-MM-DD HH:MM’ is OK, use CHAR(16) or anything similar if you cannot find any suitable type, provided that date/time functions of your RDBMS will work OK with that type. Everything other is just stored as VARCHAR.

I recommend taking types schema from mod_archive_odbc_mysql.sql and triggers creation code from mod_archive_odbc_sqlite3.sql and then adjusting it to suit your RDBMS.

Encodings

Erlang + ODBC knows nothing about encodings, so they just pass the text further to native driver without changes. As ejabberd and XMPP protocol in general use UTF-8, you should make sure that the native driver expects to receive UTF-8 from you. For most *nixes this is easy to accomplish: just configure unixODBC with charset=utf-8 or, if using native driver directly, specify encoding in your DBMS config.

For Windows, there’s one important complication: Windows treats erlang + ODBC as ANSI client, so when mod_archive_odbc connects to the database which has Unicode ODBC driver, Windows performs ANSI codepage→Unicode conversion, completely messing your UTF-8 input.

I’m not really sure what is the best & universal solution for it - at least I haven’t found the way to solve this problem for SQLite3 Unicode ODBC driver. What I’ve finally done for that was patching the driver to expose ANSI interface, so that input is passed to driver without conversion, and later treat it as UTF-8 - hacky and wrong, but works OK. Just in case, if anyone needs these, here is the patch for sqliteodbc-0.76 sources and here is the compiled binary.

For more complex / powerful RDBMS’es, such as MS SQL, as far as I understood there are the ways in ODBC driver itself to specify what encoding to assume for particular DSN when clients connect to it via ANSI interface.

Most likely, you will also want to specify UTF-8 as encoding of the database when creating it, or make sure that it’s created by default with UTF-8 encoding.

Closing thoughts

I hope that releasing mod_archive_odbc should improve server-side message archiving situation as it provides mostly standard-conformant and full-featured implementation, thus making it easier to develop clients supporting server-side message archiving.

My other project, libwsw, once brought up to fully useable state, should make it easier to add XEP-136 support to clients, once it is finished, and can already be used for simple maintenance operations like importing / exporting history to / from XEP-136 compliant servers.

Please, do not hesitate to comment!

Comment viewing options

ndl@home is currently in maintenance. During this maintenance it is not possible to change site content (like comments, pages and users).
Select your preferred way to display the comments and click "Save settings" to activate your changes.

пользуюсь mod_archive_odbc на

пользуюсь mod_archive_odbc на локалхосте, заметил проблему: когда пользователь залогинен с нескольких ресурсов, и оба эти ресурса получают одинаковое сообщение (так рассылает, например, google talk) -- то в БД, соответственно, появляются идентичные записи от разных ресурсов. Как я понял из исходников, никаких средств контроля доступа к таблице archive_messages не предусмотрено?

Identical messages

NDL's picture

Ну да, должны появляться: если контакту пришло N сообщений (неважно, на тот же ресурс или разные) то N сообщений и сохранится, а одинаковый текст или разный - это уже не модуля дело.

Здравствуйте. Установил на

Здравствуйте. Установил на ejabberd2 mod_archive_odbc. Сообщения сохраняются в MySQL. При запросе

<iq type='get' id='7825:list' xmlns='jabber:client'><list xmlns='http://www.xmpp.org/extensions/xep-0136.html' with='user1@abboom.com'/></iq>

выдается ошибка

<body xmlns='http://jabber.org/protocol/httpbind'><iq xmlns='jabber:client' from='user695044@abboom.com' to='user695044@abboom.com/8526413907' type='error' id='7825:list'><list xmlns='http://www.xmpp.org/extensions/xep-0136.html' with='user1@abboom.com'/><error code='503' type='cancel'><service-unavailable xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/></error></iq></body>

В чем может быть проблема? Куда копать?

mod_archive_odbc '503' error

NDL's picture

Добрый день,

Я бы не рекомендовал использовать mod_archive_odbc для новых инсталяций, он уже много лет как не поддерживается и у меня нет ни одной его работающей инсталяции (= я даже не знаю какая была последняя версия erlang / ejabberd / RDBMS bindings на которой он еще работал).

Я сейчас использую mod_archive2 - для персональной инсталяции он работает достаточно стабильно. Если есть возможность - я бы рекомендовал использовать именно его. Учтите, что форматы конфигов и базы у mod_archive_odbc и mod_archive2 отличаются, скрипт для миграции данных есть - но его нужно запускать для каждого аккаунта отдельно.

Если нет возможности использовать mod_archive2 и действительно нужно использовать mod_archive_odbc -то включите логгинг у ejabberd и посмотрите, в каком месте именно модуль крешится. Если не получится понять, в чем проблема - пришлите кусок лога мне (+ точная версия ejabberd, erlang и mysql erlang адаптера) попробую глянуть.

Crash om mod_archive_obdbc

i have successfully installed mysql with ejabberd 13.12 and i have also imported https://github.com/processone/ejabberd/blob/master/sql/mysql.sql and mod_archive_odbc_mysql.sql in my database. in ejabberd.cfg :

{auth_method, odbc}.
{odbc_server, {mysql, "localhost", "ejabberd", "ejabberd", "password"}}.
this works perfectly. but when i add module (mod_archive_obdc) :

{mod_archive_odbc, [{database_type, "mysql"},
{default_auto_save, true},
{enforce_default_auto_save, false},
{default_expire, infinity},
{enforce_min_expire, 0},
{enforce_max_expire, infinity},
{replication_expire, 31536000},
{session_duration, 1800},
{wipeout_interval, 86400}]},

then show following crash report.

2014-04-24 17:45:38 =CRASH REPORT====
crasher:
initial call: mod_archive_odbc:init/1
pid: <0.381.0>
registered_name: []
exception exit: {{{case_clause,undefined},[{gen_iq_handler,add_iq_handler,6,[{file,"src/gen_iq_handler.erl"},{line,65}]},{mod_archive_odbc,init,1,[{file,"mod_archive_odbc.erl"},{line,213}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]},[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
ancestors: [ejabberd_sup,<0.37.0>]
messages: []
links: [<0.306.0>]
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 27
reductions: 174
neighbours:

Please help me.
Thanks in advance

ejabberd versions > 2.x are not supported

NDL's picture

mod_archive_odbc was written for ejabberd 2.x version and, therefore, cannot run on 13.x and above.

Any suggestions for ejabberd v14.05

Just got to know that mod_archive_odbc is not for v14.05 . Any suggestions what to do for this version.

Any suggestions for ejabberd v14.05

NDL's picture

No, I’m not aware of any XEP-136 implementations for ejabberd > 2.x

Webviewer not working

I am using the current version from https://svn.process-one.net/ejabberd-modules with the Debian wheezy ejabberd version.
If I try to access the webviewer, I get the following error:

  crasher:
    initial call: ejabberd_http:init/2
    pid: <0.493.0>
    registered_name: []
    exception error: bad argument
      in operator  ++/2
         called as <<100,0,97,0,110,0,105,0,101,0,108,0>> ++ "@"
      in call from jlib:jid_to_string/1 (jlib.erl, line 243)
      in call from mod_archive_webview:'-search_form/2-fun-0-'/1 (src/mod_archive_webview.erl, line 355)
      in call from lists:map/2 (lists.erl, line 1173)
      in call from mod_archive_webview:search_form/2 (src/mod_archive_webview.erl, line 354)
      in call from mod_archive_webview:process2/3 (src/mod_archive_webview.erl, line 140)
      in call from ejabberd_http:process/2 (ejabberd_http.erl, line 339)
      in call from ejabberd_http:process_request/1 (ejabberd_http.erl, line 392)
    ancestors: [ejabberd_http_sup,ejabberd_sup,<0.37.0>]
    messages: []
    links: [#Port<0.3650>,<0.293.0>,#Port<0.3649>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 6765
    stack_size: 24
    reductions: 2115
  neighbours:

Can you help me?
Best regards

mod_archive_odbc.beam failed: badfile

Can you please make .beam files and give a link directly to them.
I have tried a thousands times to make .beam files from given processes. But there is always issues and bad .beam files are genrated.

Please give mod_archive.beam and others directly.

mod_archive_odbc.beam failed: badfile

root@ip-10-40-106-62:/opt/ejabberd-2.1.11/bin# ./ejabberdctl start
root@ip-10-40-106-62:/opt/ejabberd-2.1.11/bin# ./ejabberdctl status
Failed RPC connection to the node ejabberd@localhost: nodedown
 
Commands to start an ejabberd node:
  start  Start an ejabberd node in server mode
  debug  Attach an interactive Erlang shell to a running ejabberd node
  live   Start an ejabberd node in live (interactive) mode
 
root@ip-10-40-106-62:/opt/ejabberd-2.1.11/bin# cd ../logs
root@ip-10-40-106-62:/opt/ejabberd-2.1.11/logs# tail ejabberd.log
=ERROR REPORT==== 2013-06-07 03:04:09 ===
Loading of /opt/ejabberd-2.1.11/lib/ejabberd-2.1.11/ebin/mod_archive_odbc.beam failed: badfile
 
=ERROR REPORT==== 2013-06-07 03:04:09 ===
C(<0.37.0>:gen_mod:75) : Problem starting the module mod_archive_odbc for host "riklr.com"
 options: []
 error: undef
 
=ERROR REPORT==== 2013-06-07 03:04:09 ===
C(<0.37.0>:gen_mod:80) : ejabberd initialization was aborted because a module start failed.

I compiles files as were given in the tutorial
I am using odbc (mysql) database.
I have no idea why this "badfile" error is coming.
Above is the tail of log file.
Please help me out with this?

We need CONN_MYSQ ejabber module!

All internet tutorials about mod_archive in Ejabberd are incomplete or very caotics. Or maybe are old. So my problem is that I am using Ubuntu 12.04 server LTS with Mysql. And I did all the things that are explained in all tutorials or blogs.

So to install mod_archive I need to do this, the typical:
>> svn co https://svn.process-one.net/ejabberd-modules
>> cd ejabberd-modules/mod_archive/trunk
>> ./build.sh
>> cp ebin/*modules in EJABBERD
>> and configure /etc/ejjaberd.cfg

¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡ but with this methods EJABBERD does not work!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

So I discover that we need to compile other MODULE that are not explained in any site. The module is called CONN_MYSQL

¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡¡We need CONN_MYSQ!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

And we can get in the same ejabber-modules dowloaded. So in SVN (https://svn.process-one.net/ejabberd-modules) there are cd ejabberd-modules/mysql/trunk ./build.sh cp ebin/*modules in EJABBERD

And work !!! Fine!!! :D

I need a lot of hours to discover this :'(

I can not use this line configuration in ejabberd.cfg: {odbc_server, "DSN=ejabberd;UID=ejabberd;PWD=ejabberd"} because aplication crashes and says this:

=ERROR REPORT==== 2012-11-28 18:01:08 === E(<0.437.0>:mod_archive_odbc:867) : should_store_jid failed: {xmlelement, "error", [{"code","500"}, {"type", "wait"}], [{xmlelement, "internal-server-error", [{"xmlns", "urn:ietf:params:xml:ns:xmpp-stanzas"}], []}]}

So I use this line in ejabberd.cfg: {odbc_server, {mysql, "server", "database", "username", "password"}}.

and this line into the modules configuration: {mod_archive_odbc, [{database_type, "mysql"}, {default_auto_save, true}, {enforce_default_auto_save, true}]},

So, now works!!!!!!!!!

I don't remmember if I use

'trunk' or 'branches/ejabberd-2.0.x' in Emakefile . You could try one and if not works you could try the other.

Ah! and remmember to compile with:
./configure --enable-odbc

the ejabberd source code. You can get this in "wget http://www.process-one.net/en/ejabberd/ejabberd/2.1.11/ejabberd-2.1.11.tgz"

maybe it is necessary this packages: "apt-get install iodbc libmyodbc"

Good luck!!

mysql 'offset' error

I use strophe.archive.js to get archived messages, when I retrieving a collection, I get an error response, the following are ejabberd log:

E(<0.334.0>:mod_archive_odbc:2428) : failed transaction: {aborted,
{{case_clause,
{aborted,
"#42000You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '30offset 0' at line 1"}},
[{mod_archive_odbc,
get_messages_query,
2,
[{file,
"src/mod_archive_odbc.erl"},
{line,1883}]},
{mod_archive_odbc,
get_messages,2,
[{file,
"src/mod_archive_odbc.erl"},
{line,1856}]},
{mod_archive_odbc,
retrieve_collection_and_msgs,
2,
[{file,
"src/mod_archive_odbc.erl"},
{line,1041}]},
{mod_archive_odbc,
'-process_local_iq_retrieve/3-fun-0-',
4,
[{file,
"src/mod_archive_odbc.erl"},
{line,1032}]},
{ejabberd_odbc,
outer_transaction,
3,
[{file,
"ejabberd_odbc.erl"},
{line,396}]},
{ejabberd_odbc,
run_sql_cmd,4,
[{file,
"ejabberd_odbc.erl"},
{line,322}]},
{p1_fsm,
handle_msg,10,
[{file,
"p1_fsm.erl"},
{line,544}]},
{proc_lib,
init_p_do_apply,
3,

The correspond generated SQL is like 'select * from archive_messages where coll_id = 3 order by utc, id limit 30offset 0'. You can see there lack a space between '30' and 'offset', so I doubt the following code under mod_archive_odbc.erl has some bug(maybe lack a space after "escape(Max)", I do not know erlang ):

get_request_part_range(UTCField, {{index, Index}, Max}) ->
["order by ", UTCField, ", id ",
if Max /= undefined -> ["limit ", escape(Max)]; true -> "" end,
"offset ", escape(Index)].

Would you mind to take a look?
Best regards

Re: mysql 'offset' error

NDL's picture

Hello,

Sorry for late response - was on vacation …

Indeed, looks like there’s an error in that fragment of the code, not sure why it wasn’t exercised by other XEP-136 client implementations - but it’s an error nonetheless.

Normally I don’t support mod_archive_odbc anymore (I don’t even have it installed on my server), but in this case the fix should be simple enough to still do it, plus you’ve already done most of the work anyway (thanks for that!) I tried to submit it to ejabberd-modules SVN repo, but looks like I don’t have credentials for that anymore, so here’s the patch that might fix it, please let me know if it doesn’t:

--- src/mod_archive_odbc.erl (revision 1131)
+++ src/mod_archive_odbc.erl    (working copy)
@@ -1647,7 +1647,7 @@
 
 get_request_part_range(UTCField, {{index, Index}, Max}) ->
     ["order by ", UTCField, ", id ",
-     if Max /= undefined -> ["limit ", escape(Max)]; true -> "" end,
+     if Max /= undefined -> ["limit ", escape(Max), " "]; true -> "" end,
      "offset ", escape(Index)].

Handling of the thread attribute

I want to map all the messages between two unique jid's in the same collection and I found in the XEP-0136 manual that including a unique thread tag within the message packet is the best way to do that. It does work but after sometime when I try the same thing a new collection is created and the rest of the messages get stored in the newly created collection, even though the same unique thread is included within the new messages being sent.
It is being referred in mod_archive_odbc.erl that we should strip resouce completely from JID when creating the key. Well, I did stripped the resources from the "from" and "to" attributes of the message stanza but the same thing keeps on happening.
Is it an error on My side or is it a bug...?

session_duration?

NDL's picture

Hard to say for sure, but I’d suspect new collection is created after no activity for more than “session_duration” seconds?

Using “thread” attribute in mod_archive_odbc implementation guarantees only that even if different resources are used, messages are still stored in the same collection, but not that all messages end up in the same collection regardless of time delay between these messages.

This might be not 100% compatible to the behavior described in XEP-136 spec, but the issue is that otherwise implementation would be considerably more complex (I’d have to create separate index table to store thread → collection_id mapping and do that table lookup for all messages for which there is no collection info in memory cache), so given no real-world & widely-used clients seem to care about including ‘thread’ attribute anyway, this extra complexity doesn’t seem to be justified.

Is this issue resolved in

Is this issue resolved in mod_archive2...?

'thread' attribute handling

NDL's picture

No, as I don’t consider it to be an ‘issue’ (at least as long as major clients don’t support ‘thread’ attribute anyway)

problem with mod_archive_webview

On Debian with PSQL 8.4.12 and erland R14 mod_archive_webview have problem with received data/time from DB.

mod_archive_webview expect text format of utc time, but from DB driver is recived separated numeric format of utc time.

This quick and simply patch fix this problem in mod_archive_webview:

*** mod_archive_webview.erl.orig	2012-07-16 15:15:02.146789021 +0000
--- mod_archive_webview.erl	2012-07-17 14:11:09.142838136 +0000
***************
*** 102,108 ****
      make_xhtml(?T("Chat with ") ++ Jid, contact_config(Jid,US,Lang) ++
                             [?XE("ul", lists:map( fun({Id, Node, Server, Resource, Utc, Subject }) -> 
                                                      With = jlib:jid_to_string({Node,Server,Resource}),
!                                                     ?LI([?AC(?LINK("show/" ++ integer_to_list(Id)), "On " ++ Utc ++ " with " ++ With ++ " -> " ++ escape_str(Subject)  )] ) end,
                                                  get_collection_list(jlib:string_to_jid(Jid), US)))
                 ], Lang);
 
--- 102,111 ----
      make_xhtml(?T("Chat with ") ++ Jid, contact_config(Jid,US,Lang) ++
                             [?XE("ul", lists:map( fun({Id, Node, Server, Resource, Utc, Subject }) -> 
                                                      With = jlib:jid_to_string({Node,Server,Resource}),
!                                                     {{Utc_Y, Utc_M, Utc_D}, {Utc_H, Utc_m, Utc_s}} = Utc,
!                                                     Date = integer_to_list(Utc_Y) ++"-"++ integer_to_list(Utc_M) ++"-"++ integer_to_list(Utc_D)
!                                                             ++" "++ integer_to_list(Utc_H) ++":"++ integer_to_list(Utc_m) ++":"++ integer_to_list(Utc_s),
!                                                     ?LI([?AC(?LINK("show/" ++ integer_to_list(Id)), "On " ++ Date ++ " with " ++ With ++ " -> " ++ escape_str(Subject)  )] ) end,
                                                  get_collection_list(jlib:string_to_jid(Jid), US)))
                 ], Lang);
 
***************
*** 211,217 ****
          0 -> { jlib:jid_to_string({WithU,WithS,WithR}) , "message_from" } ;
          1 -> { jlib:jid_to_string({LUser,LServer,""}) , "message_to" } 
      end,
!     [_Date, Time] = string:tokens(Utc, " "),
      ?XAE("p", [{"class", Class}] , [ ?XAE("span", [{"class","time"}], [?C("["++Time++"]")]), ?C(" "),
                                     ?XAE("span", [{"class","jid"}], [?C(From)]), ?C(": "),
                                     ?XAE("span", [{"class","message_body"}], [?C(Body)])]).
--- 214,221 ----
          0 -> { jlib:jid_to_string({WithU,WithS,WithR}) , "message_from" } ;
          1 -> { jlib:jid_to_string({LUser,LServer,""}) , "message_to" } 
      end,
!     {{Utc_Y, Utc_M, Utc_D}, {Utc_H, Utc_m, Utc_s}} = Utc,
!     Time = integer_to_list(Utc_H) ++":"++ integer_to_list(Utc_m) ++":"++ integer_to_list(Utc_s),
      ?XAE("p", [{"class", Class}] , [ ?XAE("span", [{"class","time"}], [?C("["++Time++"]")]), ?C(" "),
                                     ?XAE("span", [{"class","jid"}], [?C(From)]), ?C(": "),
                                     ?XAE("span", [{"class","message_body"}], [?C(Body)])]).
***************
*** 459,464 ****
--- 463,471 ----
          %If the previous query fail, that mean the collection doesn't exist or is not 
          % one of the users connection.
 
+         {{Utc_Y, Utc_M, Utc_D}, {Utc_H, Utc_m, Utc_s}} = Utc,
+         UtcR = integer_to_list(Utc_Y) ++"-"++ integer_to_list(Utc_M) ++"-"++ integer_to_list(Utc_D)
+                ++" "++ integer_to_list(Utc_H) ++":"++ integer_to_list(Utc_m) ++":"++ integer_to_list(Utc_s),
          {selected, _ , List} = run_sql_query("SELECT utc,dir,body"
                                                   " FROM archive_messages"
                                                   " WHERE coll_id = '" ++ ejabberd_odbc:escape(Id) ++ "'"),
***************
*** 467,473 ****
                                      " WHERE us = " ++  SUS ++  " AND deleted=0 "
                                      "  AND with_user ='" ++ ejabberd_odbc:escape(WithU) ++ "'" ++
                                      "  AND with_server ='" ++ ejabberd_odbc:escape(WithS) ++ "'" ++
!                                     "  AND utc > '" ++ Utc ++ "'" ++
                                      " ORDER BY utc LIMIT 1") of
              {selected, _ , [{V1}]} -> V1;
              _ -> -1
--- 474,480 ----
                                      " WHERE us = " ++  SUS ++  " AND deleted=0 "
                                      "  AND with_user ='" ++ ejabberd_odbc:escape(WithU) ++ "'" ++
                                      "  AND with_server ='" ++ ejabberd_odbc:escape(WithS) ++ "'" ++
!                                     "  AND utc > '" ++ UtcR ++ "'" ++
                                      " ORDER BY utc LIMIT 1") of
              {selected, _ , [{V1}]} -> V1;
              _ -> -1
***************
*** 477,488 ****
                                      " WHERE us = " ++  SUS ++  " AND deleted=0 "
                                      "  AND with_user ='" ++ ejabberd_odbc:escape(WithU) ++ "'" ++
                                      "  AND with_server ='" ++ ejabberd_odbc:escape(WithS) ++ "'" ++
!                                     "  AND utc < '" ++ Utc ++ "'" ++
                                      " ORDER BY utc DESC LIMIT 1") of
              {selected, _ , [{V2}]} -> V2;
              _ -> -1
          end,
!         { {WithU,WithS,WithR} , Utc,  Subject , List, {PrevId,NextId}} end,
      run_sql_transaction(LServer, Fun).
 
  change_subject(Id,Subject,{LUser,LServer}) ->
--- 484,495 ----
                                      " WHERE us = " ++  SUS ++  " AND deleted=0 "
                                      "  AND with_user ='" ++ ejabberd_odbc:escape(WithU) ++ "'" ++
                                      "  AND with_server ='" ++ ejabberd_odbc:escape(WithS) ++ "'" ++
!                                     "  AND utc < '" ++ UtcR ++ "'" ++
                                      " ORDER BY utc DESC LIMIT 1") of
              {selected, _ , [{V2}]} -> V2;
              _ -> -1
          end,
!         { {WithU,WithS,WithR} , UtcR,  Subject , List, {PrevId,NextId}} end,
      run_sql_transaction(LServer, Fun).
 
  change_subject(Id,Subject,{LUser,LServer}) ->

ejjabred and mysql

hello

I have installed ejjabred an mySql on my laptop ,So i would like to configure my ejjabred to use Mysql and not default database Mnseia

how i do it

thanks in advance

Не отдает историю

Пробовал воспользоваться данным модулем mod_archive_odbc
в итоге сообщения в mysql базе появляются
а вот при попытке получить их в клиенте(Vacuum)
получаю следующее:

<<<< admin@jabber.magius.org.ua/Vacuum-IM 17:18:33 +62 <<<<

30

также пробовал
mod_archive
сообщения корректно сохраняются и доступны в клиенте.
Однако хотелось бы именно ваш вариант(mysql)

Укажите пожалуйста куда копать, спасибо.

выкинуло всю ошибку((

выкинуло всю ошибку((

list xmlns="http://www.xmpp.org/extensions/xep-0136.html#ns" end="2012-06-10T20:59:59Z" with="magius@jabber.magius.org.ua" start="2012-05-09T21:00:00Z"
set xmlns="http://jabber.org/protocol/rsm"
max xmlns="http://jabber.org/protocol/rsm"30/max
/set
/list
error xmlns="jabber:client" type="modify" code="400"
bad-request xmlns="urn:ietf:params:xml:ns:xmpp-stanzas"/
/error

More info needed

NDL's picture

Укажите, пожалуйста, следующую информацию:

  • Версию клиента (= Vacuum IM).
  • Версию ejabberd сервера.
  • Версию mod_archive_odbc.
  • Версию mysql адаптера для ejabberd.
  • Секция из ejabberd-конфига, относящаяся к mod_archive_odbc + секция настройки mysql соединения оттуда же.

У меня все работает как раз под mysql и с Vacuum, т.е. либо какие-то нестыковки по версиям, либо - по конфигурации.

> Версию клиента (= Vacuum

> Версию клиента (= Vacuum IM).
1.1.2

> Версию ejabberd сервера.
2.1.10

> Версию mod_archive_odbc.
Скачал из SVN
В шапке исходника написано
Version : 1.0.1

> Версию mysql адаптера для ejabberd.
нативный драйвер r14b3

> Секция из ejabberd-конфига, относящаяся к mod_archive_odbc + секция настройки mysql соединения оттуда же.
{mod_archive_odbc, [{database_type, "mysql"},
{default_auto_save, true},
{enforce_default_auto_save, false},
{default_expire, infinity},
{enforce_min_expire, 0},
{enforce_max_expire, infinity},
{replication_expire, 31536000},
{session_duration, 1800},
{wipeout_interval, 86400}]},

{odbc_server, {mysql, "localhost", "ejabberd", "ejabberd", "pass"}}.

я прошу прощения за назойливость, не могли бы мы перейти к общению через почту/джаббер/skype для ускорения процесса.
Честно говоря я уже не ждал что вы ответите, огромное спасибо за ответ.

mod_archive_webview does not start

Hello!

mod_archive_webview does not start for me, althogh mod_archive_odbc works OK and wites messages to DB successfully. I use PostgreSQL. The only relevant message I see in logs is

=ERROR REPORT==== 2011-01-04 14:03:11 ===
E(<0.36.0>:gen_mod:73) : {undef,
[{mod_archive_webview,start,["lvu.kiev.ua",[]]},
{gen_mod,start_module,3},
{lists,foreach,2},
{ejabberd_app,start,2},
{application_master,start_it_old,4}]}

Is it possible to determine the source of the problem? What additional information do I have to provide?

Configuration problem?

NDL's picture

What additional information do I have to provide?

  • Version of ejabberd you use.
  • Whether ODBC or native RDBMS adapter is used.
  • Part of config where mod_archive_webview is configured.

Option to force archiving

As far as I know, XEP-0136 does not has any such option. But if the module could provide an option for forcing archiving, it would be very useful for enterprises. Is there any chance that you'd consider adding such an option to ejabberd.cfg (perhaps in mod_archive2)?

Re: Option to force archiving

NDL's picture

As far as I know, XEP-0136 does not has any such option.

XEP-136 explicitly mentions this scenario: see “6. Automatic Archiving” section. But you’re right that the way to specify this option is beyond the scope of XEP-136.

mod_archive_odbc in fact allows you to enforce it: just set both enforce_default_auto_save and default_auto_save to true. However please note that this does not provide the complete solution as knowledgeable user would still be able to remove his achieved messages using remove command.

I’m unlikely to work on improving support for “enforce archiving” option in mod_archive2 in the foreseeable future: it’s close to the bottom of my priorities as I do not need it for my personal use, and my personal opinion is that any enterprise bold enough to enforce messages archiving for its employees should be prepared to either pay for complete commercial solution or hire someone to assist adjusting free software to their needs.

Of course I do not insist that everyone should agree with this point of view, it’s just the way I think about it

You probably are right in

You probably are right in that enterprises which expect a lot of functionality in free software should be willing to pay for it. But it also depends on several factors such as to what extent you want the product to be adopted by individuals and enterprises, the complexity in adding a new feature, how feature rich you want your product to be, etc.

I myself am a free software developer. I thought adding such an option (of enforcing email archiving) would provide a very nice feature for very little development effort (perhaps a few lines of code). Therefore I asked you reg. it.

But now I think the only way to do this in a simple way is to disable the remove command for the user. And this might not be acceptable as it violates the RFC.

I was just wondering if we disable deletion of rows from the archive_messages table and set the above two options, would it in a crude way help in enforcing archiving?

Re: poor man's way of achieving enforcement

NDL's picture

But now I think the only way to do this in a simple way is to disable the remove command for the user. And this might not be acceptable as it violates the RFC.

Violating spec is not a show-stopper, especially considering the fact that it might be changed if enough justifications are given, but there’s more to that than simply disabling remove command: first of all, if you want to enforce achieving for some pre-defined period of time (not forever) but also allow your users to keep some conversations for longer periods of time, there are non-trivial changes to be done in achieving preferences processing - which is quite complex even in current version of XEP-136, by the way.

Second problem is described below.

It’s possible there are even more issues that should be addressed (such as actions to be done on user removal, etc) - I’ve not analysed the whole specification to find all problematic areas, but even those issues I know already are enough to see it’s not as trivial as it might appear at first sight

I was just wondering if we disable deletion of rows from the archive_messages table and set the above two options, would it in a crude way help in enforcing archiving?

To the best of my current understanding - yes, it should allow “crude” version of enforced achieving, but note that users still might amend existing conversations with arbitrary content by uploading manually the content to the chat element or create arbitrary new conversations - according to the spec, server must add uploaded content to the existing conversation or create new one if it is not yet present, so if the goal is to have exact copy of the conversations user had - “crude” approach is not enough.

Web archive reader does not work

I tested this with ejabberd 2.1.5 on Debian Lenny. Whenever I try to perform some search from the web archive reader, I get the following error in erlang.log.

=CRASH REPORT==== 2-Sep-2010::19:58:32 ===
  crasher:
    pid: <0.601.0>
    registered_name: []
    exception error: bad argument
      in function  integer_to_list/1
         called as integer_to_list("1")
      in call from mod_archive_webview:format_search_result/2
      in call from lists:map/2
      in call from mod_archive_webview:process2/3
      in call from ejabberd_http:process/2
      in call from ejabberd_http:process_request/1
      in call from ejabberd_http:process_header/2
      in call from ejabberd_http:receive_headers/1
    initial call: ejabberd_http:init({gen_tcp,#Port<0.557>},
                                     [inet,http_bind,http_poll,web_admin,
                                      {request_handlers,
                                       [{["archive"],mod_archive_webview}]},
                                      {ip,{0,0,0,0}}])
    ancestors: [ejabberd_http_sup,ejabberd_sup,<0.41.0>]
    messages: []
    links: [<0.252.0>,#Port<0.557>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 6765
    stack_size: 23
    reductions: 7644
  neighbours:

=SUPERVISOR REPORT==== 2-Sep-2010::19:58:32 ===
     Supervisor: {local,ejabberd_http_sup}
     Context:    child_terminated
     Reason:     badarg
     Offender:   [{pid,<0.601.0>},
                  {name,undefined},
                  {mfa,
                      {ejabberd_http,start_link,
                          [{gen_tcp,#Port<0.557>},
                           [inet,http_bind,http_poll,web_admin,
                            {request_handlers,
                                [{["archive"],mod_archive_webview}]},
                            {ip,{0,0,0,0}}]]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,worker}]


=CRASH REPORT==== 2-Sep-2010::19:58:32 ===
  crasher:
    pid: <0.602.0>
    registered_name: []
    exception error: bad argument
      in function  integer_to_list/1
         called as integer_to_list("1")
      in call from mod_archive_webview:format_search_result/2
      in call from lists:map/2
      in call from mod_archive_webview:process2/3
      in call from ejabberd_http:process/2
      in call from ejabberd_http:process_request/1
      in call from ejabberd_http:process_header/2
      in call from ejabberd_http:receive_headers/1
    initial call: ejabberd_http:init({gen_tcp,#Port<0.558>},
                                     [inet,http_bind,http_poll,web_admin,
                                      {request_handlers,
                                       [{["archive"],mod_archive_webview}]},
                                      {ip,{0,0,0,0}}])
    ancestors: [ejabberd_http_sup,ejabberd_sup,<0.41.0>]
    messages: []
    links: [<0.252.0>,#Port<0.558>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4181
    stack_size: 23
    reductions: 3169
  neighbours:

=SUPERVISOR REPORT==== 2-Sep-2010::19:58:32 ===
     Supervisor: {local,ejabberd_http_sup}
     Context:    child_terminated
     Reason:     badarg
     Offender:   [{pid,<0.602.0>},
                  {name,undefined},
                  {mfa,
                      {ejabberd_http,start_link,
                          [{gen_tcp,#Port<0.558>},
                           [inet,http_bind,http_poll,web_admin,
                            {request_handlers,
                                [{["archive"],mod_archive_webview}]},
                            {ip,{0,0,0,0}}]]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,worker}]

And these are the last lines in the debug log of ejabberd.log:


=INFO REPORT==== 2010-09-02 19:58:32 ===
D(<0.287.0>:ejabberd_odbc:434) : MySQL, Send query
"SELECT coll_id,subject,with_user,with_server,with_resource,C.utc,body FROM archive_collections as C, archive_messages as M WHERE C.id = M.coll_id AND C.us = 'admin@example.net'   AND C.deleted='0' GROUP BY coll_id"


=INFO REPORT==== 2010-09-02 19:58:32 ===
D(<0.287.0>:ejabberd_odbc:434) : MySQL, Send query
"commit;"

This is one really useful module and I am consider packaging it for the official Debian repository. Keep up the great work.

Re: Web archive reader does not work

NDL's picture

Hello Rahul,

  1. I’m not the author of mod_archive_webview, it was written by Olivier Goffart. I don’t think he currently maintains this module, though, so I’ll try to give you some ideas to experiment with, but don’t expect too much.
  2. What RDBMS do you use? Do you use native driver or ODBC?

So far it looks like there might be incompatibility between the format results are returned from your RDBMS in and the format mod_archive_webview expects data in, namely - Id is returned as string, not as integer. You might test if that is the case by replacing on the 414th line of mod_archive_webview.erl the text integer_to_list(Id) with just Id.

There were three such places

There were three such places where integer_to_list(Id) had been used and I have changed all of them to Id. I can now see the listing of the messages . But when I click on the message to see the entire chat conversation, I get:

=CRASH REPORT==== 7-Sep-2010::17:21:34 ===
  crasher:
    pid: <0.423.0>
    registered_name: []
    exception error: no case clause matching "1"
      in function  mod_archive_webview:format_message/3
      in call from lists:map/2
      in call from mod_archive_webview:process2/3
      in call from ejabberd_http:process/2
      in call from ejabberd_http:process_request/1
      in call from ejabberd_http:process_header/2
      in call from ejabberd_http:receive_headers/1
    initial call: ejabberd_http:init({gen_tcp,#Port<0.511>},
                                     [inet,http_bind,http_poll,web_admin,
                                      {request_handlers,
                                       [{["archive"],mod_archive_webview}]},
                                      {ip,{0,0,0,0}}])
    ancestors: [ejabberd_http_sup,ejabberd_sup,<0.37.0>]
    messages: []
    links: [<0.256.0>,#Port<0.511>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4181
    stack_size: 23
    reductions: 6597
  neighbours:

=SUPERVISOR REPORT==== 7-Sep-2010::17:21:34 ===
     Supervisor: {local,ejabberd_http_sup}
     Context:    child_terminated
     Reason:     {case_clause,"1"}
     Offender:   [{pid,<0.423.0>},
                  {name,undefined},
                  {mfa,
                      {ejabberd_http,start_link,
                          [{gen_tcp,#Port<0.511>},
                           [inet,http_bind,http_poll,web_admin,
                            {request_handlers,
                                [{["archive"],mod_archive_webview}]},
                            {ip,{0,0,0,0}}]]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,worker}]


=CRASH REPORT==== 7-Sep-2010::17:21:34 ===
  crasher:
    pid: <0.425.0>
    registered_name: []
    exception error: no case clause matching "1"
      in function  mod_archive_webview:format_message/3
      in call from lists:map/2
      in call from mod_archive_webview:process2/3
      in call from ejabberd_http:process/2
      in call from ejabberd_http:process_request/1
      in call from ejabberd_http:process_header/2
      in call from ejabberd_http:receive_headers/1
    initial call: ejabberd_http:init({gen_tcp,#Port<0.512>},
                                     [inet,http_bind,http_poll,web_admin,
                                      {request_handlers,
                                       [{["archive"],mod_archive_webview}]},
                                      {ip,{0,0,0,0}}])
    ancestors: [ejabberd_http_sup,ejabberd_sup,<0.37.0>]
    messages: []
    links: [<0.256.0>,#Port<0.512>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 4181
    stack_size: 23
    reductions: 2964
  neighbours:

=SUPERVISOR REPORT==== 7-Sep-2010::17:21:34 ===
     Supervisor: {local,ejabberd_http_sup}
     Context:    child_terminated
     Reason:     {case_clause,"1"}
     Offender:   [{pid,<0.425.0>},
                  {name,undefined},
                  {mfa,
                      {ejabberd_http,start_link,
                          [{gen_tcp,#Port<0.512>},
                           [inet,http_bind,http_poll,web_admin,
                            {request_handlers,
                                [{["archive"],mod_archive_webview}]},
                            {ip,{0,0,0,0}}]]}},
                  {restart_type,temporary},
                  {shutdown,brutal_kill},
                  {child_type,worker}]

What could be the reason for this? Unfortunately I am not an Erlang developer and I cannot make out much from the error. Your help is really appreciated.

Re: There were three such places

NDL's picture

That seems to be roughly the same kind of a problem: try changing case Dir of at line 210 to case list_to_integer(Dir) of

Bravo! Bravo! A big round of

Bravo! Bravo! A big round of applause for you. That fixed the error. Is this a bug specific to mysql? If not, would you consider making the changes upstream? Below is the patch.

--- mod_archive_webview.erl.orig	2010-09-09 17:43:08.000000000 +0530
+++ mod_archive_webview.erl	2010-09-09 17:38:16.000000000 +0530
@@ -102,7 +102,7 @@
     make_xhtml(?T("Chat with ") ++ Jid, contact_config(Jid,US,Lang) ++
                            [?XE("ul", lists:map( fun({Id, Node, Server, Resource, Utc, Subject }) -> 
                                                     With = jlib:jid_to_string({Node,Server,Resource}),
-                                                    ?LI([?AC(?LINK("show/" ++ integer_to_list(Id)), "On " ++ Utc ++ " with " ++ With ++ " -> " ++ escape_str(Subject)  )] ) end,
+                                                    ?LI([?AC(?LINK("show/" ++ Id), "On " ++ Utc ++ " with " ++ With ++ " -> " ++ escape_str(Subject)  )] ) end,
                                                 get_collection_list(jlib:string_to_jid(Jid), US)))
                ], Lang);
 
@@ -207,7 +207,7 @@
 %------------------------
 
 format_message({ Utc, Dir, Body } ,{WithU,WithS,WithR}, {LUser,LServer} ) ->
-    {From, Class} = case Dir of 
+    {From, Class} = case list_to_integer(Dir) of 
         0 -> { jlib:jid_to_string({WithU,WithS,WithR}) , "message_from" } ;
         1 -> { jlib:jid_to_string({LUser,LServer,""}) , "message_to" } 
     end,
@@ -411,7 +411,7 @@
 
 format_search_result( {Id,Subject,User,Server,Resource,Utc,Body} ,_Lang) ->
     ?XAE("p",[{"class","search_result"}],
-         [?AC(?LINK("show/" ++ integer_to_list(Id)), jlib:jid_to_string({User,Server,Resource}) ++ " : " ++ escape_str(Subject)),
+         [?AC(?LINK("show/" ++ Id), jlib:jid_to_string({User,Server,Resource}) ++ " : " ++ escape_str(Subject)),
           ?C(Body), ?XE("em",[?C(Utc)]) ] ).
 
 links_previous_next({PrevId,NextId},Lang) ->
@@ -422,7 +422,7 @@
 links_previous_next_aux(Class, Text, Id) ->
     case Id of
         -1 -> [];
-        _ -> [?XAE("a",[{"href",?LINK("show/" ++ integer_to_list(Id))},{"class",Class}], [?C(Text)])]
+        _ -> [?XAE("a",[{"href",?LINK("show/" ++ Id)},{"class",Class}], [?C(Text)])]
     end.
 
 %------------------------

There is at least one client

There is at least one client supporting xep-0136 now - it's called Vacuum IM (crossplatform). Got working installation with mod_archive_odbc + Vacuum IM as xep-0136 capable client.

Client support

There is at least one client supporting xep-0136 now - it's called Vacuum IM (crossplatform). I was happy to move to my own ejabberd server with server-side history - thanks to mod_archive_odbc AND working server-side history capable client.

Vacuum IM

NDL's picture

Yes, in fact I use this client now too and exactly due to the same reason

postgresql error in expire_collections()

I noticed some SQL errors while checking postgres logs.
I'm using mod_archive_odbc (ejabberd-modules r1065), ejabberd 2.1.3 and PostgreSQL.
I think that the propper query could look like this:

2189c2189
- "pgsql" -> ["timestamp archive_collections.", UTCField, " + interval ", ExpExpr, " || ' seconds'"];
+ "pgsql" -> ["archive_collections.", UTCField, " + interval '", ExpExpr, " seconds'"];

Re: postgresql error in expire_collections()

NDL's picture

Yes, that’s very well possible it is broken, as I haven’t tested mod_archive_odbc with PostgreSQL myself, I’ve got only single brief success report from one user who tried it but no information whether all features were working as they should or not (including collections expiration).

Just for information: mod_archive_odbc is not developed anymore, its replacement is mod_archive2. It includes PostgreSQL support from the start and its automatic tests are run on all supported RDBMSes, including PostgreSQL, to ensure things work correctly for all of them. It’s for ejabberd 3.x only and not yet finished, though.

As for the particular way to fix it: current code in mod_archive2 comparison expression construction looks like this (that’s just part of the whole expression of course):

get_expired_condition(RDBMS, "full_jid_prefs.expire", UTCField), " < ", Now

where get_expired_condition is defined as

get_expired_condition(RDBMS, ExpireField, UTCField) ->
    case RDBMS of
        mysql ->
            ["timestampadd(second, ", ExpireField, ", ", UTCField, ")"];
        sqlite ->
            ["datetime(", UTCField, ", '+' || ", ExpireField, " || ' seconds')"];
        pgsql ->
            [UTCField, " + (", ExpireField, " || ' seconds')::interval"];
        _ ->
            throw({error, 'internal-server-error'})
    end.

It’s possible your suggestion might also work, but I haven’t tested it and don’t remember the PostgreSQL syntax to do it by heart, the only thing I remember is that it appeared unnecessarily complex and counter-intuitive to me

A-ha. Thank you for the info

A-ha. Thank you for the info and thank you for mod_archive2(_odbc)

retrieve error code

Hi,

I have the same problem ( Paolo Nesti ) when I send a packet "list" I have no problem but when I send a packet "retrieve" i have this erro message :

<iq xmlns='jabber:client' from='karim@localhost' to
='karim@localhost/39545700811236769946286949' type='error'><retrieve xmlns='http://www.xmpp.org/extensions
/xep-0136.html#ns' exactmatch='1' with='thomas@localhost'><set xmlns='http://jabber.org/protocol/rsm'
/></retrieve><error code='400' type='modify'><bad-request xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'
/></error></iq>

and log message :

=INFO REPORT==== 2009-03-11 12:08:05 ===
D(<0.292.0>:ejabberd_router:301) : route
from {jid,"karim","localhost",[],"karim","localhost",[]}
to {jid,"karim","localhost","1224605941236767674859589",
"karim","localhost","1224605941236767674859589"}
packet {xmlelement,"iq",
[{"type","error"}],
[{xmlelement,"retrieve",
[{"xmlns",
"http://www.xmpp.org/extensions/xep-0136.html#ns"},
{"exactmatch","1"},
{"with",
"thomas@localhost/5948674921236766883956325"}],
[{xmlelement,"set",
[{"xmlns","http://jabber.org/protocol/rsm"}],
[]}]},
{xmlelement,"error",
[{"code","400"},{"type","modify"}],
[{xmlelement,"bad-request",
[{"xmlns","urn:ietf:params:xml:ns:xmpp-stanzas"}],
[]}]}]}

my config :

- ejabberd 2.0.2_
- Mysql

But I haven't problem for store message in db.

Have you a idea ? the problem it's a ressource ?

The JID must be full JID

NDL's picture

Hi Karim,

It looks like XML error packet does not correspond to the log message, no? (resources seem to be completely different …)

If your “retrieve” command is indeed sent with parameter “with=’thomas@localhost”’ - then yes, the problem must be that you specify bare JID instead of full JID, which is not allowed by XEP-136 - see discussion with Paolo below for details.

bare jid

Thanks,

But I use Strophe library and ressource is always different ?

karim

mod_archive_odbc+LDAP auth

Hi, is there a way of using mod_archive_odbc when auth type is LDAP. It works fine when I use %{auth_method, odbc} with native mysql driver, but when change to %{auth_method, ldap}, I get this error:
** Generic server 'ejabberd_mod_archive_odbc_ja.mydomain.ru' terminating
** Last message in was {'$gen_cast',
{addlog,"chat",to,"roshchin.d","ja.mydomain.ru",
"Home",
{jid,"root","ja.mydomain.ru","QIP","root",
"ja.mydomain.ru","QIP"},
[],[],[],"hyh"}}
** When Server state == {state,"ja.mydomain.ru",
{dict,0,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[]},
{{[],[],[],[],[],[],[],[],[],[],[],[],[],
[],[],[]}}},
1800}
** Reason for termination ==
** {noproc,{gen_server,call,
['ejabberd_odbc_sup_ja.mydomain.ru',which_children,
infinity]}}

mod_archive_odbc+LDAP auth

P.S. my ejabberd is 1.1.4, maybe it is fixed in 2.0?

Re: mod_archive_odbc+LDAP auth

NDL's picture

Yes, most likely your problem is caused by 1.x version usage - see details in the article, search for “However, for ejabberd-1.x versions the following needs to be done” pattern

If your setup permits, I would recommend upgrade to 2.x, though.

exact match in list and retrieve

Thank you for this module it's great to have the chat history in a database.
I'm working at a web client that will be release as gpl and I'm currently struggling with a feature that cannot make work as stated in XEP-136. Honestly I'm not sure if it's me doing or expecting something wrong or if actually there is this discrepancy:

When I list chats I can retrieve them without specifying a resource identifier in the with attribute, and ejabberd will return all chats with that JID.
This goes even if I use the exactmatch="1"

The query:
<iq xmlns="jabber:client" type="get" id="get_chats">
<list xmlns="http://www.xmpp.org/extensions/xep-0136.html#ns" with="jeff@example.com" start="2008-01-01T00:00:00.000000Z"><set xmlns="http://jabber.org/protocol/rsm"><max xmlns="">100</max>
</set></list></iq>

will give me:
<iq xmlns='jabber:client' from='paolo@example.com' to='paolo@example.com/__ponderaid__app__' id='get_chats' type='result'>
<list xmlns='http://www.xmpp.org/extensions/xep-0136.html#ns'>
...
<chat with='jeff@example.com' start='2008-11-26T11:07:00.000000Z'/>
<chat with='jeff@example.com' start='2008-11-27T00:25:50.000000Z'/>
<chat with='jeff@example.com/__ponderaid__app__498' start='2008-11-27T09:51:02.000000Z'/>
<chat with='jeff@example.com/__ponderaid__app__' start='2008-11-27T11:33:36.000000Z'/>
<chat with='jeff@example.com/__ponderaid__app__' start='2008-11-27T11:52:52.000000Z'/>
<set xmlns='http://jabber.org/protocol/rsm'>
<first index='0'>63393645860@65</first>
<last>63395005972@231</last>
<changed>2008-11-27T11:53:03.000000Z</changed>
<count>11</count></set></list></iq>

This is actually not a problem to me as it is the behaviour I need.

However when I try and retrieve one chat from the list, I cannot retrieve it unless I specify the resource identifier in the 'with' attribute.

If I do:

<iq xmlns="jabber:client" type="get" id="get_chat">
<retrieve xmlns="http://www.xmpp.org/extensions/xep-0136.html#ns" exactmatch="0" with="jeff@example.com" start="2008-11-27T11:52:52.000000Z">
<set xmlns="http://jabber.org/protocol/rsm"/>
</retrieve></iq>

I won't get the last chat on the list above.

The server answer will be:
<iq xmlns='jabber:client' from='paolo@example.com' to='paolo@example.com/__ponderaid__app__713' id='get_chat' type='error'>
<retrieve xmlns='http://www.xmpp.org/extensions/xep-0136.html#ns' exactmatch='0' with='jeff@example.com' start='2008-11-27T11:52:52.000000Z'>
<set xmlns='http://jabber.org/protocol/rsm'/>
</retrieve><error code='404' type='cancel'>
<item-not-found xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
</error></iq>

unless I use the JID as with="jeff@example.com/__ponderaid__app__"

Now this is different from how I read xep-0136 in may respects. Is this possibly the expected behaviour? And do you kow if there is something I can do to avoid specifying the resource in my query for a single chat?

Tia, Paolo

OOps, I should apparently

OOps, I should apparently have escaped the xml in the post above. I hope it is understandable, if not tell me I will do something to send a readable version of teh queries

/p

Re: exact match in list and retrieve

NDL's picture

OOps, I should apparently have escaped the xml in the post above. I hope it is understandable, if not tell me I will do something to send a readable version of teh queries

That’s OK, don’t worry - your original content is there, it’s just sanitized on rendering only, I’ve corrected it to read as you’ve intended it to be, if you don’t mind.

Answering your questions:

This goes even if I use the exactmatch=”1”

Yes, it is known limitation: I was one who asked to add this option to the specs, yet mod_archive_odbc does not support it yet

I’m going to bring eventually mod_archive_odbc in accordance to the current version of XEP-136 (including changing namespace, support for exactmatch, some fixes to replication and collections versioning), but I’m afraid I cannot give exact date when it happens: the “lower bound” is beginning of the next year, but for now I wouldn’t hold your breath on that.

However when I try and retrieve one chat from the list, I cannot retrieve it unless I specify the resource identifier in the ‘with’ attribute.

In fact, I believe this behavior is strictly in accordance with XEP-136, see “7.2 Retrieving a Collection” section - “The ‘with’ and ‘start’ attributes specify the participating full JID and the start time” (the emphasis is mine).

The thing is that the meaning of “list” and “retrieve” are essentially different: while “list” is used to “match” some subset of all existing collections, “retrieve” is used to get exactly one’s collection content - and that’s impossible without constraining all aspects of that collection, such as full JID and exact “start” attribute, which, according to XEP-136, uniquely identify collection. If you know a bit of *NIX API, think about it as “list” being similar to “opendir/readdir” (get list of all files), while “retrieve” is “open/read” (read content of single file).

And do you kow if there is something I can do to avoid specifying the resource in my query for a single chat?

I’m afraid with current XEP-136 version there’s just no way you can both match & retrieve collection in single step However, during my experiments with client implementation it didn’t seem that important: in most cases you still have to retrieve & present somehow the list of collections to the client - and while doing that, you get full JIDs for collections you need anyway.

If you describe your case in more detail (I mean the reason you need “retrieve” to perform matching), probably I could try to comment on that/suggest some other solution - though, probably, you might do it better than me anyway

Post new comment

ndl@home is currently in maintenance. During this maintenance it is not possible to change site content (like comments, pages and users).
Syndicate content