Vibe and WebDAV
This is a summarization of my thoughts about the topic “Vibe and WebDAV”. It represents my personal views and is based on my knowledge and my own tests and experiences. It turned out that to open the subject matter to a wide audience, it needs to be be a longer read. However, quite a bit of the information included here may be very helpful in troubleshooting the topic at hand.
Links to product pages and further information are collected at the bottom of this entry.
The Entities/Players and their relations
This section gives an overview of the entities and player in this field. I see this as necessary information to put any discussion below into the right context.
Novell Vibe as a collaboration platform has come a long way from the portlet-based Teaming+Conferencing solution, currently boosting the efficiency-augmenting functionality by integrating with the Windows Desktop and with MS Office and beginning to see a growing customer base that “gets” where the benefits of Vibe lie – in being a powerful, enterprise-grade “framework application” that can be tailored to local needs without the need for and the lock-in of expensive design consulting like it is seen with other software in the same flexibility range.
WebDAV has been around for a long time, providing file operations via HTTP requests, and thus being able to cross firewalls and proxies transparently. With HTTPS, the transfer is transparently secured end-to-end, and concurrent access is regulated with locking capabilities. WebDAV extensions are in widespread use (like CalDAV/GroupDAV/DAV-SVN), but “plain” WebDAV as a networked file access method is not very common, thought also not rare. Where more traditional file transfer protocols like FTP, SCP, SMB, CIFS and such hit their respective barriers, WebDAV is often a viable alternative. In general, however, end-users often have little or no experience with WebDAV.
Microsoft Windows’ treatment of WebDAV directly reflects the end-user perspective – the usage of general, standards-based WebDAV is not transparently documented and has a high number of pitfalls and depends strongly not only on the version of Windows, but also on the flavor (32 vs. 64 bit). The client software included in Microsoft Windows is easily able to (and seems geared to) connect to Microsoft’s WebDAV solutions like SkyDrive ans SharePoint. To get WebDAV working for mapping arbitrary WebDAV resources into WinXP 32 bit to Win7x64, the admins or users “arsenal” needs to hold a rather wide range of hotfixes and registry changes, and still, success may vary a lot with the server one is trying to talk to. With Windows 7, security measures in the WebDAV client software have been stepped up, and by default, an SSL connection using a valid, trusted (by Windows) and matching-name certificate is mandatory, as is the use of non-basic authentication. When failing to connect, the user gets no messages, but usually only another login prompt – which can in rare cases lead to user lock-out on the target system. Another big issue with using WebDAV on Microsoft Windows is the rather short maximum path length of a WebDAV URL, paired with Windows using “http[s]://<user>@<fully.qualified.domain.name>:<port>/” in the URL, thereby bloating the path and making error tracking harder by giving disadvantages to users with longer user names.
Apple Mac OS X incorporates WebDAV support into its “Finder” natively since 10.0, https support and basic authentication support, however, were introduced in 10.4. Apple has used WebDAV as an access method for their – by now discontinued – iDrive product, and according to the reports of end users, the implementation of the WebDAV client in Mac OS X seems to be geared to connecting to Apple-developed WebDAV servers, and connections to arbitrary WebDAV servers often show unpredictable results, the most extreme being the crashing of the Mac OS X Finder application.
Linux based clients have the most stable and widest ranging support for the WebdAV protocol included. This coincides with the higher affinity to technology and higher system knowledge generally found in users of Linux systems (end users working on managed Linux desktop systems benefit from the knowledge of their admin and the functionality and flexibility of the built-in libraries). WebDAV-wise, Linux systems benefit highly from the Open Source paradigm “if it doesn’t work for you, help us fix it”, giving weight to taking care of niche scenarios and more exotic server implementations instead of focusing development effort on a monolithic, customer-lock-in type solution.
Microsoft Office is – in general – able to work on WebDAV connected shares. However, to work with certain WebDAV server implementations, only certain combinations of Windows versions, flavors (32/64) and Office versions are known to work. Additionally, some parts of Microsoft Office behave differently from most programs, in that they present the user one single document, but create a folder and several files in the target location.Additionally, the abovementioned WebDAV access for SharePoint and SkyDrive seems to be handled in a way different from the one Windows uses. To use a “web location” to save to, this location needs not only to implement standards-compliant WebDAV and have security properties similar to the ones described above, but also needs to implement several SOAP interfaces, presenting a high hurdle to jump for third-party server products.
LibreOffice has a built-in, transparent WebDAV client implementation, which I have personally not worked with, yet. Some searching shows issues not being extremely frequent.
Third-Party applications are very varied in their focus, approach and level of support of WebDAV. There are several widely used applications, such as BitKinex and CyberDuck, and usually, WebDAV is only one of a large number of protocols they support.
With this taken care of, I will now focus on the relationship between Vibe and WebDAV, and on the necessity of having WebDAV support in Vibe.
WebDAV in Vibe
Vibe uses WebDAV in several places, some of them “behind the scenes”. The implementation of WebDAV is based on the Jakarta Slide project (more on that below). Going from the backend to the frontend, there are the following instances.
- WebDAV as a backend for mirrored folders
- WebDAV as a method for interfacing with attachments used by the “Edit in Place” applet
- WebDAV as a method of accessing attachments stored in Vibe
As a backend for mirrored folders, I have tried using WebDAV a few times, but never to any extent that would warrant a well-founded opinion. I am looking into using WebDAV to “mirror” file folders between Vibe sites/zones, and also to mirror other WebDAV-enabled directories, e.g. parts of SVN repositories. Since WebDAV is not as widespread as it could be, the priorities of my projects have unfortunately been lying elsewhere. And especially because I have seen so many issues with the WebDAV implementation in Vibe working as a client, I was not eager to put an overly big amount of effort into this.
From a server admin perspective, WebDAV-backed mirrored folders enable a wide range of options for getting files into Vibe and still have a secure firewall setup.
In the edit in place functionality, WebDAV mostly stays in the background, invisible to the end user, and producing a low number of errors or incidents. However, there are some situations where the limitations of Windows’ WebDAV support come into play. Again, the biggest nuisance here is the “silent” failure (i.e. showing the login box again) in case of an error. The low number of issues found when using this way of communicating with Vibe via WebDAV could stem from the use of the “/internal” WebDAV part (see below) combined with URL-escaping spaces and other special characters. More about that, also below.
As a server endpoint for WebDAV connections, Vibe directly faces the end user or other systems working as clients, and needs to operate with a wide variety of client implementations and operating systems. For providing WebDAV functionality, Vibe relies on the “Jakarta Slide” Java library, provided by the Jakarta project of the Apache Software Foundation. This setup has some issues that I will get to below, but on the whole, it has worked for the use cases and scenarios in the install base of Novell Teaming/Vibe OnPrem. With the release of Office 2010 and Windows 7×64, and with the interesting mix of issues and conditions inherent there, however, the WebDAV functionality of Vibe has shown to become less usable and more error-prone.
Issues of the Vibe/Slide setup
As mentioned above, Vibe uses Jakarta Slide in the backend, and the web frontend of the “/ssfs/” web application shows “Jakarta Slide 2.2” being used. However, the last official release of the Slide project was 2.1 on December 26th, 2004, and, quoting the Jakarta Project news page, “Due to the lack of a developer community, the code base was no longer actively maintained and security issues could not be addressed by bugfix releases. The Jakarta PMC therefore had no other choice but to retire Slide”. So Vibe uses a non-official release of Slide, probably built on the sources of Slide-2.2pre. The WebDAV working group at the W3C, however, concluded work on WebDAV in March of 2007. Without analyzing Slide further, it can be said that Slide probably deviates quite a bit from what could be called “WebDAV standards compliant”, not to mention the possibility of security issues still hidden inside.
As a result of that, there is a host of issues seen with connecting to Vibe via WebDAV. In the release notes of Vibe 3.2, Section “7.19 WebDAV Issues” lists six distinct issues, covering close to two A4 pages in print. There are several bugs listed in Novell’s bug tracker, and my personal experience has shown some more interesting issues, most of them of course “not only” coming from Vibe, but rather from the interaction between Vibe and the different clients used.
A rather high percentage of issues out there seem to have some connection with spaces in URLs or with too long URLs. As for binders having spaces in WebDAV URLs, Vibe is “special”, since in a default Vibe installation, that unfortunately is every binder, leading me to have “renaming the top three workspaces to remove spaces” as a first task on any installation.
Some of the bigger issues I (and others) came across, in addition to the ones listed in the places just mentioned, are:
- On Mac OS X, it is not possible to connect directly to a binder of which the URL has a blank space in the path. The issue has been tracked to coming from Mac OS X escaping the special character ” ” twice, getting to “%20” and then “%2520” by escaping the “%”. Removing the spaces helps a lot here. However, that only works for reading via WebDAV. On writing a file via WebDAV, I have crashed the Finder app repeatedly. This seems to come from the Finder first saving the “resource fork” (link below) to WebDAV, leading the server to regard the transaction as finished, and then starting to push the “real” data, but choking on the missing of a reply from Vibe. This is handled properly by almost all WebDAV implementations that have been in active development around the Mac OS X 10.5 time frame (2006).
- On iOS, a massive majority of the apps I have tried out do not allow spaces in URLs. Often, this is done by removing the space bar from the on-screen keyboard and can be overcome by copy/pasting the URL, and then, WebDAV seems to work with the third-party apps, but apps by Apple (like Pages, Keynote, etc) still fail when trying to work over WebDAV.
- Windows only allows a certain length of the URL for WebDAV connections. Some Office products have extra ones (e.g. OneNote 2010, having 150 characters). This can get messy, especially if users work in their personal workspaces. Consider the following: let my account name be “christian.giese” (it is, at some customer installations), and let the Vibe server have the URL vibe.internal.customersite.co.uk and running SSL on port 8443(also not uncommon). Then, the URL of a file called “x.jpg” in my personal File folder is
https://firstname.lastname@example.org:8443/ssfs/files/library/Home Workspace/Personal Workspaces/Christian Giese (giese)/Files/x.jpg
. Counted until the last slash, this gives me a hefty 145 characters to start from. This can get ugly rather quickly.
- Some applications – independent of the OS used – speak their own dialects and accents of WebDAV. Other server products have kept pace with that. Slide, of course, hasn’t. A current big issue is with OneNote 2010, where working with WebDAV seems to need so much work to get functional that it can be called “virtually impossible”. This unfortunately pushes customers to using Microsoft SharePoint, since it “just works”.
One might ask “why do these problems do not occur with the edit in place functionality? It’s using WebDAV, too!”. This comes from the kind of URL that the Java applet uses to access the files. As mentioned above, it uses another endpoint to access the attached files: “/ssfs/files/internal” instead of “/ssfs/files/library”. This means that the URL path of a file given file is based not on its position in the binder hierarchy, but on the id of the binder and the id of entry it is attached to, so the sample URL above changes like this
, with <Bnd> and <Ent> counting up with system size, seldom reaching 6-digit numbers. With 5-digit IDs for both, I get 95 characters length for a base URL, and this is why edit in place hits problems a lot later and a lot more seldom. In addition to that, all special characters inside the URL that the applet uses are escaped, so it comes without spaces or other hiccup-inducers in the attachment file name.
So what is the problem with using that endpoint? This endpoint was never designed for end user interaction, but for use with one-shot actions, like edit in place and like sending a single attachments WebDAV URL to someone. This manifests in three areas: navigation (and folder manipulation), readability, and scalability.
- Navigating this structure is close to impossible for a user, since the hierarchy of binders is not exposed. Mapping a network location to such a URL makes only one folder visible, all its children/subfolders are invisible. Also, creating folders via this structure is sure to wreak havoc and go wrong (I am afraid to even try that). And seeing OneNote 2010 creating folders without asking/informing the user means that there are more pitfalls around than anyone can live with.
- Readability of URLs becomes very important as soon as users are exposed to them, but it does not need to be “fully there” in many cases. If a user uses his personal file folder, they want to see their name in the URL; also, if a user needs to navigate a structure, binder IDs – even if they would be shown in hierarchy – do not help. However, if a user is sent a WebDAV link by mail or has a drive mapped for him by the workstation administrator, then the path readability steps to the background, again.
- Scalability comes into play as soon as a system gets big in terms of binder/item count. With the URL scheme above, imagine browsing that structure via WebDAV. I have done – and timed – that on a system with over 30.000 entries and some 10.000 binders. Showing the URL /ssfs/internal/files”, which is the list of binders, took 232 seconds (just under 4 minutes) to open and loaded the server quite a bit during that time. So scalability of this endpoint is not good.
As can easily be seen, WebDAV with Vibe works, in a way. There are many issues, though, that can be countered with experience, knowledge, and quite some work. This situation is bearable for seasoned veterans with a high pain threshold, and can be handled with a lot of docs and support load, but for the “general public”, it is not something to run to.
So why use WebDAV at all?
Why Vibe Needs WebDAV Support
With Vibe Desktop and the Microsoft Office Add-In about to be released in early 2012, what impact does WebDAV still have? Is it worth putting in development effort to get it working as smoothly and transparently as it should be? Being a beta tester for the two products mentioned, I say “Yes, definitely so!”.
Vibe Desktop is currently only available for Microsoft Windows (a Mac OSX version is scheduled for mid-2012). It does a wonderful job and can replace the (often hated) use of DropBox inside organizations, giving users, managers and admins peace from the nightmares about data loss and information leakage. However, it synchronizes files, making them available for offline use (with a limit on file size, fortunately). It does not access the Vibe Server “live”, enabling browsing through the folder structure and opening files directly.
The MS Office Add-In is an absolute productivity booster. It is exactly the kind of stuff that Vibe needs to satisfy and to “wow” the customer. Direct integration into the “productivity suite” is the maximum you can get in the area of being seamless. However, it is only available for Microsoft Office and not for LibreOffice, OpenOffice, StarOffice, TextMaker Office, it is only available on Windows, not on Mac OS X or Linux. Excluding all these users can outweigh all the positive effects of the great functionality. Another issue is that it only works with Word, Excel, and PowerPoint. The more collaborative tools like OneNote and Outlook are not supported, even though there are great opportunities, there, like enabling a filing of Outlook items into Vibe for further work (not necessarily by drag and drop like in GroupWise, but by button/ribbon, like it is now in Word etc.).
WebDAV is a standards-based protocol, and having WebDAV capability in Vibe represents the only way of accessing, browsing, and working with information inside the Vibe structure without having to design, develop, build, test, provide and support client software for all flavors and kinds of clients and OSs. Advanced usage, like synchronizing Vibe binders across zones/sites via WebDAV and such, will keep admins happy and productive.
Improving Vibe’s WebDAV
The core issue for Vibe is – from my perspective – the outdated implementation of WebDAV in use. Slide is, by all means, dead and should be buried. Some time spent searching and some analysis have led to Bugzilla reports (one linked below) and to quite some posts on beta forums (alas, now closed to me), including the finding of a possible replacement library for Slide, called Milton. I do not have enough time to spare to really dig deeply into it, but it seems powerful, and it has a good-looking client compatibility list. I am aware that this involves some development work, and a quick scan of the source of Kablink Vibe shows that this might be not completely limited to rewriting mapping classes and interfaces. Still, what is it worth, getting Mac OS X on board and shortening the Readme document by two pages? There might be other, better alternatives that I am not aware of, and I would appreciate any and all information on that.
The secondary issues from a technical side seem to be the length of URLs and the URLs having “forced spaces” in them. Both of these issues could be taken care of by a sort of proxy servlet/mapping not unlike like the one that listens to the URL context “/vibe” and delivers the short URLs for folders and workspaces. What users could work with and would love, is a functioning, short WebDAV URL along the lines of “http[s]://<teamingsite>/filebox/<userLogonName>”, providing a “FileBox” (I don’t know if “DropBox” is trademarked globally) folder in the personal workspace without switching to the “ugly”, long URL listed above – that’s why I call it “proxy servlet”.
This could then be used for a load of added functionality, like providing folders via WebDAV using the short-URL name that is configured for the folder, like “http[s]://<teamingsite>/ShortDAV/<FolderShortName>”, implying that “http[s]://<teamingsite>/ShortDAV/<FolderShortName>/<subfolder1>/<subfolder1.1/>/<filename>” works, too. That would be extremely helpful, especially since this way, users can create their own, personal “WebDAV shares” with memorable URLs including full access control leverage to share files with others. This has the power to take a big load off of administration staff – and it could tie in with other products, like Filr.
[On a related note, this proxy servlet could be used to enable shorter permalinks, like:
- “http[s]://<teamingsite>/pl/e/<EntryID>” for an entry permalink
- “http[s]://<teamingsite>/pl/b/<BinderID>” for a folder/workspace permalink
- “http[s]://<teamingsite>/pl/u/<UserLogonName>” for a permalink to a users profile or workspace (configurable by the admin)
If spaces in the path keep being a problem after replacing Slide (or after fronting it with a smart proxy servlet taking care of that), these spaces in file and binder names can be countered by URL-escaping them, like it is done now already by/for the applet, so this would be a UI job for the places where this URL is presented.
The Bottom Line
For me, the bottom line of this analysis is:
Vibe needs WebDAV, and needs it working smoothly and easily. Please, Novell, make it happen.
Feedback, comments, questions, additional information… anything and everything is welcome and appreciated.
Code and Concept
WebDAV on Wikipedia: http://en.wikipedia.org/wiki/WebDAV
The Jakarta Slide project: http://jakarta.apache.org/slide/
The Jakarta Slide news page: http://jakarta.apache.org/slide/news.html
The Jakarta Slide-2.2pre source: http://svn.apache.org/repos/asf/jakarta/slide/trunk/
Vibe 3.2 Readme: http://www.novell.com/documentation/vibe32/vibe32_readme_novell/data/vibe32_readme_novell.html
Mac OS Resource Fork on Wikipedia: http://en.wikipedia.org/wiki/Resource_fork
Milton, a possible Slide replacement: http://milton.ettrema.com
The “Milton” bug in Novell’s Bugzilla: https://bugzilla.novell.com/show_bug.cgi?id=697687