Sep 29 14:06:35 hey everyone Sep 29 14:06:43 who's here? Sep 29 14:07:04 * racke waves Sep 29 14:07:06 >chanserv< op #interchange Sep 29 14:07:07 --- ChanServ sets modes [#interchange +o pj] Sep 29 14:07:16 im here Sep 29 14:07:33 --- pj has changed the topic to: Interchange - The e-commerce platform || http://www.icdevgroup.org/ || IRC Meeting now in progress. Sep 29 14:08:42 is that all? I was sure we had more confirmed to be here than the three of us Sep 29 14:09:37 FWIW, I'm here Sep 29 14:09:44 hehehe, ok Sep 29 14:10:14 I think David was going to be here, and Jon was going to be running late. Sep 29 14:10:32 anyways, we may as well get started ... Sep 29 14:11:49 Shall we just jump right into it? First item on the agenda (these are in no particular order, btw, so we can jump around) is * Review of the recent release, bug reports, etc. Sep 29 14:12:26 We just did a release on all current branches. How did it go and what bugs, if any, came up that still need to be addressed? Sep 29 14:12:53 We may also want to revisit this a bit when Jon gets back because he's release manager. Sep 29 14:13:51 Pass. I don't have any live catalogs, so it wasn't urgent for me to roll out the new release. And I didn't make time to test it. Sorry. Sep 29 14:14:16 yes Sep 29 14:14:30 David says he might be also later and will leave after an hour ... Sep 29 14:14:57 daniel_b: are you around Sep 29 14:15:01 Personally I think it actually went very well and while there was a couple of bugs that I found, they were not related to the security fixes which is the main thing I was worried about. I did run across an issue later on with some old deep links from another site, but that was to be expected with this type of release and was not a bug in IC. Sep 29 14:15:03 we have some dev sites up and running on 5.6.2 and everythign seems ok so far... Sep 29 14:15:03 Yep Sep 29 14:15:24 my clients sites are running 5.7.2, btw. Sep 29 14:15:57 yeah, release went quite ok or nobody cared so much, take your pick :-) Sep 29 14:16:04 hehehe Sep 29 14:16:11 well, I think it actually went very smoothly Sep 29 14:16:59 <--jeff_b (n=jeff_b@216.163.43.209) has left #interchange Sep 29 14:17:09 we will be gradually moving all our clients over from 5.2 & 5.4 to 5.6.2 so will let you know if any issue arises Sep 29 14:17:20 I haven't rolled out my 5.6.2 test site so don't have antyhing to report on that front Sep 29 14:17:25 although, from certain posts I've seen to the ml it would seem that some people don't seem to be upgrading right away and would prefer to take their chances with the security vuln. Sep 29 14:17:39 yes, as always Sep 29 14:17:41 I did upgrade a site from 5.2 to 5.6.2 without any major issues including all of their little hacks. Sep 29 14:17:48 no surprise here Sep 29 14:17:52 fenic: cool Sep 29 14:18:13 did we ever get the patch out for older versions? Sep 29 14:18:16 it's also not in production yet but I'm just waiting on them to sign off Sep 29 14:21:32 I don't think we did. We should get that out and available fo people who do not wish to upgrade. Sep 29 14:22:18 nope Sep 29 14:22:38 5.2 is more than five years old Sep 29 14:22:55 yeah, but there is a note in the UPGRADE file saying that a patch is available. Sep 29 14:23:19 * endpoint_david here now Sep 29 14:23:24 ...reviewing logs Sep 29 14:23:24 Hiya David Sep 29 14:23:35 hey pj Sep 29 14:23:47 We've just been discussing the recent release and how it went. Sep 29 14:24:08 where is the note, pj? Sep 29 14:24:22 If you do not wish to upgrade your Interchange installation to fix this Sep 29 14:24:22 vulnerability, patches for all currently supported Interchange versions are Sep 29 14:24:22 also available from http://www.icdevgroup.org/. You will still need to Sep 29 14:24:22 follow the upgrade advice contained here. Sep 29 14:24:27 at the very end of the file Sep 29 14:24:45 well then 5.2 isn't currently supported Sep 29 14:25:15 right, but we did say we would release a patch, I should not have said, "older versions" Sep 29 14:25:31 although the patch should apply to older versions with varying levels of success. Sep 29 14:26:05 everyone can supply such a patch and test it, but I don't think it is worth our voluntary time Sep 29 14:26:32 well then we should remove that note from the UPGRADE file and anywhere else it might appear. Sep 29 14:28:21 sure Sep 29 14:28:55 hello all Sep 29 14:28:59 Hiya jon_jensen Sep 29 14:29:09 we were just discussing the recent release, anything to add? Sep 29 14:29:24 nope, I agree with Racke that anything < 5.4 is not supported Sep 29 14:29:34 we can tweak the UPGRADE note so it'll appear in the next release Sep 29 14:29:43 ok Sep 29 14:29:54 then let's move on ... Sep 29 14:30:07 * Remaining UTF-8 issues that have to be dealt with before we release 5.8. Sep 29 14:30:22 The main blocker to an IC 5.8 release is a few UTF-8 issues, imo. We should discuss which of these bugs still needs resolving and try to get them hashed out right away, or assigned to be resolved. Sep 29 14:30:42 I think racke knows the most about that, and David. Sep 29 14:30:48 most important one is the Encode::Alias problem Sep 29 14:31:03 endpoint_david: can you show that patch again? Sep 29 14:31:21 and also good would be to properly encode mail headers Sep 29 14:31:52 yes, but I think the latter is less of an issue. What is the Encode::Alias problem? Sep 29 14:32:26 well, I don't like to get flagged by spamassassin Sep 29 14:32:34 fair enough Sep 29 14:33:47 hm seems to be missing in RT Sep 29 14:34:07 one point where it triggers is Sep 29 14:34:37 File::readfile Sep 29 14:34:46 if ($encoding) { Sep 29 14:34:47 local $PerlIO::encoding::fallback = Encode::PERLQQ(); Sep 29 14:34:47 binmode(READIN, ":encoding($encoding)"); Sep 29 14:34:47 } Sep 29 14:34:53 oh that's right, I think David did have a patch for it. Sep 29 14:35:12 inside Safe Encodes requires and Safe barfs Sep 29 14:35:15 http://github.com/machack666/interchange/commit/9005f439862fd8d2deb9b3ca9849e0daf0a75d16 Sep 29 14:35:25 right, although I would like to see an enhancement Sep 29 14:35:29 racke: did you get a chance to test that in production? Sep 29 14:35:46 I have the notes for testing if we're in a safe container, but haven't implemented yet Sep 29 14:36:05 in development :-), but it works Sep 29 14:36:26 i.e., when it's ok to got to binmode($fh, ":encoding(utf8)") (which does validation) vs binmode($fh, ":utf8"), which does not Sep 29 14:36:46 the first form is what triggers the Safe exception Sep 29 14:36:47 than I tried to run the first half only if we are within Safe but that doesn't work Sep 29 14:37:02 ok, so you tried that Sep 29 14:37:53 there's also the issue of selecting the correct encoding in writefile, but some of that is dependent on the purpose of the file, particularly because content-type header isn't available to determine that in general. Sep 29 14:38:15 I tried Sep 29 14:38:19 if ($MVSAFE::Safe Sep 29 14:38:32 but that didn't work inside Safe Sep 29 14:39:08 yeah, if that's not imported into the container, it wouldn't be defined (I believe) Sep 29 14:39:27 but there may be other ways of detecting that we're in a safe container, such as looking at the package name Sep 29 14:39:43 hm but this if is used all over the place Sep 29 14:40:38 so maybe Vend::Safe doesn't use it? Sep 29 14:41:26 Vend::Safe may not, although it returns real Safe objects; it's a convenient wrapper to make sure that everything gets initialized correctly for the container Sep 29 14:42:04 we can look at reval calls to see where code is actually executed for hints on where things might be getting messed up Sep 29 14:42:54 I can easily check that one Sep 29 14:43:33 which, reval is called twice in the initialization phase for the new Safe container, so it could be dieing there :-) Sep 29 14:44:18 why should that matter? Sep 29 14:45:04 I don't know that it does, but $MVSafe::Safe isn't set at that point Sep 29 14:45:35 (if it should be) Sep 29 14:46:30 hmm, is $safe->share_from() cumulative? Sep 29 14:47:20 my $pkg = 'MVSAFE' . int(rand(100000)); Sep 29 14:47:20 undef $MVSAFE::Safe; Sep 29 14:47:20 $ready_safe = new Vend::Safe $pkg; Sep 29 14:47:20 $ready_safe->share_from('MVSAFE', ['$safe']); Sep 29 14:47:27 don't know Sep 29 14:47:33 right, lib/Vend/Interpolate.pm: $ready_safe->share_from('MVSAFE', ['$safe']); Sep 29 14:48:26 docs don't say, but I would have thought so. Sep 29 14:49:17 maybe move this code over into Vend::Safe? Sep 29 14:50:35 internal package lookup won't work as expected: The code can only see the compartment’s namespace (as returned Sep 29 14:50:35 by the root method). The compartment’s root package appears to Sep 29 14:50:36 be the "main::" package to the code inside the compartment. Sep 29 14:50:48 or at least as proposed earlier by me :-) Sep 29 14:51:10 that means? Sep 29 14:51:51 we can't look at the current package as a substitute for checking $MVSAFE::Safe Sep 29 14:52:01 it sees it as if it's main, not MVSafe123:: Sep 29 14:52:18 aha Sep 29 14:52:57 there are other places where Safe containers are used in the codebase, which is why I introduced the full-blown abstraction to begin with; at least a half-dozen or so Sep 29 14:53:39 there are other places where Safe containers are invoked? Sep 29 14:53:42 (not used) Sep 29 14:54:14 git grep -e 'Vend::Safe' --and -e new Sep 29 14:54:28 not sure Sep 29 14:55:33 s/used/created/ in my above statement :-) Sep 29 14:55:54 well, my time is short today, so maybe there are other things to look at right now Sep 29 14:56:49 yeah Sep 29 14:57:18 any other high-priority utf8 bugs? there's the one that pj had a basic fix for Sep 29 14:57:25 yeah ... Sep 29 14:57:43 the mime_name one Sep 29 14:57:54 that's related to writefile, and the fact that we don't know what the target encoding should really be, much less whether something is binary or text Sep 29 14:58:05 oh, and that one too Sep 29 14:58:07 did my suggestion on that work, using ->can? Sep 29 14:58:17 yes, it did Sep 29 14:58:36 I'm currently using this patch, but it's not the proposed one ... Sep 29 14:58:41 http://github.com/pajamian/interchange/commit/7faecf107c6c069a1028ab542ec6c825bb88e54f Sep 29 14:59:24 this is the proposed one which I haven't had time to test yet (you have to actually take both patches together, I don't know how to make github display that way): http://github.com/pajamian/interchange/commit/e46352c9e44b9e4388cacff23e68a4f416b422e9 Sep 29 14:59:46 maybe "return 'UTF-8' if $encoding eq 'UTF-8'" should do a trivial normalization for /^utf-?8/$i ? Sep 29 15:00:25 look at the second patch Sep 29 15:00:46 the /-STRICT/ seems a little too hacky to me; basically, that's still a normalization for utf8, just we're not calling it out explicitly :-) Sep 29 15:01:25 and is case sensitivity an issue wrt character encoding names? Sep 29 15:01:35 yeah, but I've checked various encodings, UTF-8 is the only one that has that problem, and it always just adds -strict to what is otherwise the correct encoding Sep 29 15:01:39 (thinking the uc $enc->name) Sep 29 15:02:00 well... Sep 29 15:02:13 yeah, there's a difference between utf8 and utf-8 which determines the exact semantics Sep 29 15:02:36 basically I compared several encodings and what mime_name and name returns Sep 29 15:02:40 sounds crazy Sep 29 15:02:43 (one is looser in its allowable input representations) Sep 29 15:02:55 sounds really crazy Sep 29 15:03:13 kind of a wart in the design, I believe. don't know if that's perl's implementation detail or something inherited from the UC Sep 29 15:03:21 I'm just munging the return result of name so it gives us the same thing that mime_name would return if it were supported in that version of Encode. Sep 29 15:03:51 have you looked at how mime_name works in newer versions of Encode? Sep 29 15:03:56 so I wanted to make sure the case was the same as well, because ... well better sage than sorry. Sep 29 15:03:58 yes Sep 29 15:04:07 probably just a hash lookup :-) Sep 29 15:04:14 well, I haven't looked at the source of mime_name but I've run tests. Sep 29 15:05:28 in any case, I think a utf8-specific codepath wouldn't be any worse in that case than any other place in the codebase. I personally prefer the regexp short-circuit approach before calling find_encoding. Sep 29 15:06:09 but I can be convinced otherwise :-) Sep 29 15:06:20 sorry, got distracted, heh Sep 29 15:07:02 one less method call = one less place for them to break our app with changes to Encode.pm :-) Sep 29 15:07:22 well, the general idea behind this method is to get any other case where it might add the -strict before that actually happens. Sep 29 15:08:30 we should be able to verify if there are any other encodings where this happens prior to the introduction of mime_name Sep 29 15:09:04 it looks like Encode uses a sub-module internally for mime_name Sep 29 15:09:55 ...and I would not be surprised to find out that the sub-module, which contains the hash lookup, isn't included in current version of Encode Sep 29 15:09:57 ...but Sep 29 15:10:09 yeah, there's a lot of stuff going on behind the scenes there; a circular dependency with Encode and Encode::Alias is what's causing the current woes with Safe + require Sep 29 15:10:11 we can just dump the hash table somewhere in IC and use it ourselves Sep 29 15:11:03 ok, so some patch, TBD, that resolves that issue Sep 29 15:11:03 just looking at it ... it appears there are a few other differences as well Sep 29 15:11:06 http://cpansearch.perl.org/src/DANKOGAI/Encode-2.37/lib/Encode/MIME/Name.pm Sep 29 15:11:10 anything else a showstopper? Sep 29 15:11:51 I can take that hash table, just dump it right into Interchange somewhere, and only use it if Encode doesn't support mime_name ... Sep 29 15:11:53 ah, nice Sep 29 15:11:58 no, but we need to fix more than the showstoppers :-) Sep 29 15:12:18 if Encode does support mime_name then we can use that so taht we get future changes to the table as well Sep 29 15:12:20 is there some way to tag tickets in RT? Sep 29 15:12:31 something based on that approach seems useful Sep 29 15:12:37 (to pj) Sep 29 15:12:42 tag like what? Sep 29 15:12:50 yeah, there is #268 Sep 29 15:12:52 UTF8-related Sep 29 15:12:55 other than the subject line Sep 29 15:13:12 but we discussed that already, I think Sep 29 15:13:24 there is a Tags extension Sep 29 15:13:28 for RT Sep 29 15:14:06 yes, we discussed that Sep 29 15:14:55 maybe looking at all places where writefile is invoked and determining where was can assume particular charsets and where we need one explicitly provided Sep 29 15:15:01 uploads should always be raw Sep 29 15:16:00 the current patch isn't going to be the best way (at least as written in the ticket), but it's hard to specify intended semantics in general without reviewing all the myriad of call sites (or callers of callers, etc). (ugh) Sep 29 15:16:07 no Sep 29 15:16:23 we get an encoding with an upload, we should use it. Sep 29 15:17:35 the current patch is just me making it work again for my client Sep 29 15:17:37 like in the post parsing phase? Sep 29 15:17:41 it is by no means the best way. Sep 29 15:17:46 err, POST parsing phase Sep 29 15:18:04 yeah, I think we can make that work if there's a header Sep 29 15:18:23 definitely for uplods, but there may be other places where writefile isn't going to do the correct thing Sep 29 15:18:29 yeah, file uploads are encoded as multipart-alternative (I think, or something like that) ... each part has it's own header with an encoding and mime type. Sep 29 15:18:52 ok, then that's doable Sep 29 15:19:25 so if content-type =~ text w/ charset, decode, otherwise leave raw Sep 29 15:19:30 that said, it needs to be fixed in core if at all possible Sep 29 15:19:41 otherwise we still have a regression for old catalogs. Sep 29 15:19:44 -->pjordan (n=pjordan@99-167-95-48.lightspeed.irvnca.sbcglobal.net) has joined #interchange Sep 29 15:19:51 Hiya Paul Sep 29 15:20:08 Hi Sep 29 15:20:22 endpoint_david: exactly Sep 29 15:20:31 hello Paul Sep 29 15:20:39 Hi Stefan Sep 29 15:20:51 thing is, I don't really get the code well enough to do that. I was barely able to get that crappy patch that makes it just work for now. Sep 29 15:20:59 :-) Sep 29 15:21:21 don't worry, I think I can make that approach work; I was just thinking at the output phase, rather than something being borked at the input phase Sep 29 15:21:44 well, I think the approach to take would be along these lines ... Sep 29 15:22:35 get the encoding at input if need be encode to utf8 at that time Sep 29 15:22:52 then the scalar that holds the file data will be utf8 encoded Sep 29 15:23:44 when writing it out we write out in utf8 ... if the utf8 flag is turned on (so the if_utf8() check should remain there) Sep 29 15:23:55 the one in my patch, I mean Sep 29 15:24:00 right Sep 29 15:24:27 ok, going to have to leave like now, will leave up the window Sep 29 15:24:27 so that means my patch is actually correct, it's just not complete, heh Sep 29 15:24:30 :-) Sep 29 15:24:32 ok Sep 29 15:24:34 ttyl Sep 29 15:24:37 ok Sep 29 15:24:45 ok, we should move on in the agenda ... Sep 29 15:25:07 * Requirements for a 5.8 release. Sep 29 15:25:07 What else needs to be done before we can get 5.8 out the door? Sep 29 15:25:49 I'm not aware of anything else that really needs to be done. Sep 29 15:26:12 are any core members still here besides you Peter? Sep 29 15:26:16 yes Sep 29 15:26:19 ok Sep 29 15:26:39 racke even said hello to you :-P Sep 29 15:26:44 ...and Jon is here Sep 29 15:26:48 i don't count him Sep 29 15:27:48 we should take the chance and "fix" the configuration semantics for UTF8 Sep 29 15:27:59 no good idea to use variables instead of configs Sep 29 15:28:14 before 5.8 Sep 29 15:29:38 hrmmmmm, yes, I agree, we can easily tell people in the UPGRADE file that they can do this in their catalog.cfg to get backwards compatibility to the var (assuming the directive is named UTF8): UTF8 __MV_UTF8__ Sep 29 15:30:29 although, the downside is we have a stable release that supports the variables Sep 29 15:31:06 yeah, but Sep 29 15:31:23 1. 5.8 already is quite different in UTF-8 handling anyway Sep 29 15:31:48 2. even with great care of BC people don't upgrade so often, so why can't they following UPGRADE notes? Sep 29 15:31:55 2. Anyone who is seriously using UTF8 is probably not using 5.6, they are probably onto 5.7 Sep 29 15:32:21 I don't think we should worry about configuration backward compat in 5.8 ... as racke said, that's what UPGRADE is for Sep 29 15:32:31 besides, it's probably going to be a very easy change Sep 29 15:32:32 sure, but these should know they can't expect compat to devel branches Sep 29 15:32:35 ok, I just thought I'd point that out Sep 29 15:32:42 right Sep 29 15:32:57 those on devel branches should know how to deal with it anyways. Sep 29 15:33:07 that's fair, gives up a tentative conclusion on that Sep 29 15:33:07 yeah Sep 29 15:33:17 and we scratch Perl* directives Sep 29 15:33:29 the way I do it is to simply diff the old UPGRADE file with the new, then I get a nice list of changes that I need to do without having to read through all the old ones. Sep 29 15:34:20 I think the Perl* directives should stick around for a bit until we're sure that they are no longer needed. I didn't see that conclusion reached in the discussion on the ml Sep 29 15:34:59 yeah, I don't see any need to rush to remove the Perl* directives ... but it would be nice to get rid of them before 5.8 if we are sure they aren't needed anymore Sep 29 15:34:59 ...and I don't see the harm in keeping them around for a bit longer, it's not like we have to encourage people to use them, or even tell people they exist. Sep 29 15:39:52 so in conclusion, racke and david will continue to work on the utf8 issues and bug #268, plus you'll switch UTF8 to a config directive, I'll churn out a better fix for #304 and jon will have his work cut out for him when we're ready for release? Sep 29 15:40:23 yes, seems so Sep 29 15:40:34 when do we meet again, in two weeks? Sep 29 15:40:54 ummmm, what day would that be? Sep 29 15:41:17 Oct 13 Sep 29 15:41:39 I don't think that will be good for me, let me check the calendar, hang on ... Sep 29 15:41:59 Sorry I was late - I'll be on time for the next one Sep 29 15:42:34 nope, I can't make it that day Sep 29 15:42:48 you're welcome to do the meeting without me. Sep 29 15:43:15 since so much revolves around UTF-8 at this point, maybe racke & david should set the next date so they can both be there? Sep 29 15:43:28 that sounds good Sep 29 15:44:05 how about racke and david work together to get those things done (and I'll pitch in where I can) they both are here regularily anyways so I don't think they need a specific meeting time... Sep 29 15:44:20 ok fair enough Sep 29 15:44:22 and then when we've made some progress we'll set a meeting date to discuss release? Sep 29 15:44:35 +1 Sep 29 15:45:18 if things seem to be going stale after a month or so I can always call a meeting to get us back on track. Sep 29 15:45:55 alright, then Sep 29 15:46:09 that concludes the meeting. I'll post the log up in a few minutes. Sep 29 15:46:42 thanks a lot & good night Sep 29 15:46:48 goodnight racke Sep 29 15:46:50 thanks, bye Sep 29 15:52:50 <--rsiddall (i=189705da@gateway/web/freenode/x-qvvpibxhrdyqrozu) has quit ("Page closed")